KOSIS openAPI 호출을 위한 AWS Lambda 생성

 

 

 

앞에서 만든 python을 AWS Lambda에 등록해본다.

https://whiseung.tistory.com/entry/KOSIS%EA%B5%AD%EA%B0%80%ED%86%B5%EA%B3%84%ED%8F%AC%ED%84%B8-%EB%8C%80%EC%9A%A9%EB%9F%89%ED%86%B5%EA%B3%84%EC%9E%90%EB%A3%8C-openAPI-%EC%82%AC%EC%9A%A9

KOSIS(국가통계포털) 대용량통계자료 openAPI 사용

 

 

Lambda 함수 생성

lambda 서비스 화면으로 이동하여, 함수를 생성한다.

 

 

 

timeout 시간 변경

기본 타임아웃 설정이 3초라, 10초 정도로 변경해준다.

 

 

코드에 작성한 함수 붙여넣기

import json
import io
import boto3
import pandas as pd
from urllib.request import urlopen
from datetime import datetime

ACCESS_KEY = 'access_key'
SECRET_KEY = 'secret_key'

def getApiUploadS3(apiUrl, s3Url, s3Path, tempFileName):
    with urlopen(apiUrl) as url:
        json_file = url.read()
        py_json = json.loads(json_file.decode('utf-8'))

        data = []
        for i, v in enumerate(py_json):
            if i == 0 :
                print(f"Title : {v['TBL_NM']}")

            value = []
            value.append(v['PRD_DE'])
            value.append(v['C1_NM'])
            value.append(v['ITM_NM'])
            value.append(v['DT'])
            value.append(v['UNIT_NM'])

            data.append(value)

    df = pd.DataFrame(data)
    df.columns = ['yyyyymm', 'category', 'item', 'value', 'unit']

    df.to_parquet(s3Url + s3Path + tempFileName
                , engine='pyarrow'
                , index=False
                , storage_options={"key": ACCESS_KEY,
                                   "secret": SECRET_KEY }
                ) 

    return s3Path + tempFileName

def lambda_handler(event, context):
    apiUrl = "<https://kosis.kr/openapi/statisticsData.do?method=getList&apiKey=api_key&format=json&jsonVD=Y&userStatsId=whiseung/115/DT_11523_400/2/1/20221123094346&prdSe=M&newEstPrdCnt=10>"
    bucketName = 'bucket_name'
    s3Path = 'parquet/'
    
    thisTime = datetime.today().strftime('%Y%m%d%H%M%S')
    tempFileName = f'KOSIS_DT_11523_400_{thisTime}.parquet'
    
    resultFileName = getApiUploadS3(apiUrl, f's3://{bucketName}/', s3Path, tempFileName)
    print(f'###  Parquet file uploaded! {resultFileName}')

    return {
        'statusCode': 200,
        'body': json.dumps('Success!')
    }

 

 

Deploy 후 Test

 

 

테스트를 위한 이벤트를 생성한다.

 

 

 

 

테스트를 실행하면, pandas 패키지가 없다는 에러 발생한다.

 

 

 

pandas Layer를 Lambda에 추가

Lambda 아래의 Layer를 클릭한다.

 

[Add a layer] 를 클릭하여 Layer를 추가한다.

pandas는 AWS layer로 제공한다.

 

 

 

 

다시 테스트

이번에는 fsspec 이 없다.

 

내 PC에서 패키지를 다운받아, 올린다.

업로드할때 파일 구성은

Labrary.zip > python > pacakges

 

c:\\workspace\\api\\lib\\python>dir
 C 드라이브의 볼륨에는 이름이 없습니다.
 볼륨 일련 번호: 1ACD-57FC

 c:\\workspace\\api\\lib\\python 디렉터리

2022-12-01  오후 04:58    <DIR>          .
2022-12-01  오후 04:59    <DIR>          ..
               0개 파일                   0 바이트
               2개 디렉터리  234,113,568,768 바이트 남음

c:\\workspace\\api\\lib\\python>pip3 install fsspec -t . --no-user
Collecting fsspec
  Using cached fsspec-2022.11.0-py3-none-any.whl (139 kB)
Installing collected packages: fsspec
Successfully installed fsspec-2022.11.0
WARNING: You are using pip version 21.1.1; however, version 22.3.1 is available.
You should consider upgrading via the 'C:\\Users\\서휘승\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\\python.exe -m pip install --upgrade pip' command.

c:\\workspace\\api\\lib\\python>dir
 C 드라이브의 볼륨에는 이름이 없습니다.
 볼륨 일련 번호: 1ACD-57FC

 c:\\workspace\\api\\lib\\python 디렉터리

2022-12-02  오전 08:37    <DIR>          .
2022-12-01  오후 04:59    <DIR>          ..
2022-12-02  오전 08:37    <DIR>          fsspec
2022-12-02  오전 08:37    <DIR>          fsspec-2022.11.0.dist-info
               0개 파일                   0 바이트
               4개 디렉터리  234,099,306,496 바이트 남음

c:\\workspace\\api\\lib\\python>

 

c:\\workspace\\api\\lib\\python>pip3 install s3fs -t . --no-user
Collecting s3fs
  Using cached s3fs-2022.11.0-py3-none-any.whl (27 kB)
Collecting fsspec==2022.11.0
  Using cached fsspec-2022.11.0-py3-none-any.whl (139 kB)
Collecting aiohttp!=4.0.0a0,!=4.0.0a1
  Using cached aiohttp-3.8.3-cp38-cp38-win_amd64.whl (324 kB)
Collecting aiobotocore~=2.4.0
  Using cached aiobotocore-2.4.1-py3-none-any.whl (66 kB)
Collecting wrapt>=1.10.10
  Using cached wrapt-1.14.1-cp38-cp38-win_amd64.whl (35 kB)
Collecting aioitertools>=0.5.1
  Using cached aioitertools-0.11.0-py3-none-any.whl (23 kB)
Collecting botocore<1.27.60,>=1.27.59
  Using cached botocore-1.27.59-py3-none-any.whl (9.1 MB)
Collecting async-timeout<5.0,>=4.0.0a3
  Using cached async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
Collecting aiosignal>=1.1.2
  Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting yarl<2.0,>=1.0
  Using cached yarl-1.8.1-cp38-cp38-win_amd64.whl (56 kB)
Collecting attrs>=17.3.0
  Using cached attrs-22.1.0-py2.py3-none-any.whl (58 kB)
Collecting charset-normalizer<3.0,>=2.0
  Using cached charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
Collecting frozenlist>=1.1.1
  Using cached frozenlist-1.3.3-cp38-cp38-win_amd64.whl (34 kB)
Collecting multidict<7.0,>=4.5
  Using cached multidict-6.0.2-cp38-cp38-win_amd64.whl (28 kB)
Collecting typing_extensions>=4.0
  Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Collecting python-dateutil<3.0.0,>=2.1
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting urllib3<1.27,>=1.25.4
  Using cached urllib3-1.26.13-py2.py3-none-any.whl (140 kB)
Collecting jmespath<2.0.0,>=0.7.1
  Using cached jmespath-1.0.1-py3-none-any.whl (20 kB)
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting idna>=2.0
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Installing collected packages: six, multidict, idna, frozenlist, yarl, urllib3, typing-extensions, python-dateutil, jmespath, charset-normalizer, attrs, async-timeout, aiosignal, wrapt, botocore, aioitertools, aiohttp, fsspec, aiobotocore, s3fs
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
boto3 1.26.17 requires botocore<1.30.0,>=1.29.17, but you have botocore 1.27.59 which is incompatible.
Successfully installed aiobotocore-2.4.1 aiohttp-3.8.3 aioitertools-0.11.0 aiosignal-1.3.1 async-timeout-4.0.2 attrs-22.1.0 botocore-1.27.59 charset-normalizer-2.1.1 frozenlist-1.3.3 fsspec-2022.11.0 idna-3.4 jmespath-1.0.1 multidict-6.0.2 python-dateutil-2.8.2 s3fs-2022.11.0 six-1.16.0 typing-extensions-4.4.0 urllib3-1.26.13 wrapt-1.14.1 yarl-1.8.1
WARNING: Target directory c:\\workspace\\api\\lib\\python\\fsspec already exists. Specify --upgrade to force replacement.
WARNING: Target directory c:\\workspace\\api\\lib\\python\\fsspec-2022.11.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: You are using pip version 21.1.1; however, version 22.3.1 is available.
You should consider upgrading via the 'C:\\Users\\서휘승\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\\python.exe -m pip install --upgrade pip' command.

c:\\workspace\\api\\lib\\python>

다운받은 라이브러리들을 압축하여 s3에 올린다.

 

 

 

 

Layer 신규 생성

좌측 메뉴에서 [계층]으로 이동한다.

계층 생성 으로 이동한다.

 

 

새로 생성한 Layer를 Lambda에 추가

 

재실행

성공

 

 

Cloud watch 에서 로그보기

 

 

s3에서 확인

 

 

 

 

다음엔 해당 Lambda를 스케쥴링 걸어보겠다.