Docker Project 1-Docker environment, build, volume, logs

 

1. Configure environement with package manages 

- Create conda environment

$ conda create -n practice_pandas

- Activate conda environment

$ conda activate practice_pandas
(practice_pandas) $

 

2. Install dependencies and generate requirements.txt file

- Install libraries

$ pip install decouple

- Generate requirements.txt file

pandas==1.2.4
elasticsearch==7.15.0
python-decouple==3.5

 

3. Create Dockerfile

FROM python:3.9.7

WORKDIR /usr/app 

COPY ./requirements.txt .

RUN pip install --no-cache-dir --upgrade pip

RUN pip install -r requirements.txt

COPY ./src/env_practice.py .

RUN mkdir result # Under the WORKDIR

CMD ["src/env_practice.py"]

ENTRYPOINT ["python3"]

 

4. Build and run

- Docker build

$ docker build -t myenv:1.0 .
[+] Building 3.1s (12/12) FINISHED              

...

- Docker run

$ docker run myenv:1.0 env_practice.py
5 10 15 20 25 30

 

5. Configure python to read value from environment. 

6. And run container with envoronment variable.

- First way is run with env-file.

# <First> Create .env file in local.

$ vi .env

ELASTIC_HOST=###.###.##.###
ELASTIC_PORT=####
ELASTIC_USER=
ELASTIC_KEY=

# <Second> Configure Python to read value from environment

import pandas as pd
import ssl
from elasticsearch.connection import create_ssl_context
from elasticsearch import Elasticsearch
import urllib3
from decouple import config

ssl_context = create_ssl_context()
ssl_context.check_hostname = False
ssl_context.verify_mode = ssl.CERT_NONE

ELASTIC_HOST=config('ELASTIC_HOST')
ELASTIC_PORT=config('ELASTIC_PORT')
ELASTIC_USER=config('ELASTIC_USER')
ELASTIC_KEY=config('ELASTIC_KEY')

es = Elasticsearch(hosts=[{'host': 'ELASTIC_HOST', 'port': 'ELASTIC_PORT'}], scheme="http",verify_certs=False, timeout=300, ssl_context=ssl_context, http_auth=("ELASTIC_USER", "ELASTIC_KEY"))
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)


# <Third> Copy .env file from my local to the container with using volume and then run.

$ docker run -v /Users/joohyunyoon/Documents/practice/.env:/usr/app/.env myenv:1.0 env_practice.py

- Second way is define the variable and its value when running the container. 

# <First> Define os module in Python code.

import pandas as pd
import ssl
from elasticsearch.connection import create_ssl_context
from elasticsearch import Elasticsearch
import urllib3
import os

ssl_context = create_ssl_context()
ssl_context.check_hostname = False
ssl_context.verify_mode = ssl.CERT_NONE

ELASTIC_HOST=os.getenv('ELASTIC_HOST')
if ELASTIC_HOST == None:
    ELASTIC_HOST='http://localhost'
ELASTIC_PORT=os.getenv('ELASTIC_PORT')
if ELASTIC_PORT == None:
    ELASTIC_PORT=9200
else:
    ELASTIC_PORT=int(ELASTIC_PORT)
ELASTIC_USER=os.getenv('ELASTIC_USER')
if ELASTIC_USER == None:
    ELASTIC_USER='user'
ELASTIC_KEY=os.environ.get('ELASTIC_KEY')
if ELASTIC_KEY == None:
    ELASTIC_KEY='123'
    
es = Elasticsearch(hosts=[{'host': ELASTIC_HOST, 'port': ELASTIC_PORT}], scheme="http",verify_certs=False, timeout=300, ssl_context=ssl_context, http_auth=(ELASTIC_USER, ELASTIC_KEY))
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

# <Second> Run container with environment variable.

$ docker run --env ELASTIC_HOST=@@@.@@@.@@.@@@ --env ELASTIC_PORT==@@@@ --env ELASTIC_USER=@@@ --env ELASTIC_KEY='@@@' myenv:1.0 env_practice.py

>>>
@@@.@@@.@@.@@@
@@@@
@@@
@@@

 

- Third way is define and get environment variables in Python then you can just use the os module.

# <First> Define and get environment variables in Python.

import pandas as pd
import ssl
from elasticsearch.connection import create_ssl_context
from elasticsearch import Elasticsearch
import urllib3
import os

ssl_context = create_ssl_context()
ssl_context.check_hostname = False
ssl_context.verify_mode = ssl.CERT_NONE

ELASTIC_HOST=os.getenv('ELASTIC_HOST')
if ELASTIC_HOST == None:
    ELASTIC_HOST='http://localhost'
ELASTIC_PORT=os.getenv('ELASTIC_PORT')
if ELASTIC_PORT == None:
    ELASTIC_PORT=9200
else:
    ELASTIC_PORT=int(ELASTIC_PORT)
ELASTIC_USER=os.getenv('ELASTIC_USER')
if ELASTIC_USER == None:
    ELASTIC_USER='user'
ELASTIC_KEY=os.environ.get('ELASTIC_KEY')
if ELASTIC_KEY == None:
    ELASTIC_KEY='123'
    
es = Elasticsearch(hosts=[{'host': ELASTIC_HOST, 'port': ELASTIC_PORT}], scheme="http",verify_certs=False, timeout=300, ssl_context=ssl_context, http_auth=(ELASTIC_USER, ELASTIC_KEY))
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

# <Second> Build the Docker file.

$ docker build -t myenv:1.0 .
[+] Building 3.1s (12/12) FINISHED                                              

...

 => => naming to docker.io/library/myenv:1.0                               0.0s
 
# <Third> Run with environment file

$ docker run --env-file .env myenv:1.0 env_practice.py

>>>
@@@.@@@.@@.@@@
@@@@
@@@
@@@
  TW_ATT_IP_SEARCH_DATA ACCD_CHARGER_ID  ... DRULE_DESC TW_DMG_IP_SEARCH_DATA

...

 

7. Python script generates output file.

import itertools
import sys
 
output=[]
for i in itertools.count(5,5):
    if i==45:
        break
    else:
        print(i,end=' ')
    output.append(i)

# <The result has been generated as an output file named result>

file_path = './result/output.txt'
sys.stdout = open(file_path, "w")
print(output)

 

8. Run container with mounted volume to access generated file.

# <First> Run container with mounted volume to access generate file.

$ docker run -v /Users/joohyunyoon/Documents/practice/result:/usr/app/result myenv:1.0 env_practice.py
5 10 15 20 25 30 [5, 10, 15, 20, 25, 30]

 

9. Redirect container output to local file instead of output to console.

$ docker logs great_pike > log.l # logs <container name> <log file name what I want>

 

10. Run container in detached mode, and after execution query the container's logs.

# <First> Run container in detached mode.

$ docker run -v /Users/joohyunyoon/Documents/practice/result:/usr/app/result -d myenv:1.0 env_practice.py
52ec2f9cb236ab68578261ee7400b3c4b8ec6517a1cdefc37598788483148b28

# <Second> Query the container's logs.

$ docker logs 52ec2f9cb236ab68578261ee7400b3c4b8ec6517a1cdefc37598788483148b28
5 10 15 20 25 30 [5, 10, 15, 20, 25, 30]