KOSIS openAPI 호출을 위한 AWS Lambda 생성

AWS

KOSIS openAPI 호출을 위한 AWS Lambda 생성

whistory 2022. 12. 9. 13:18

앞에서 만든 python을 AWS Lambda에 등록해본다.

https://whiseung.tistory.com/entry/KOSIS%EA%B5%AD%EA%B0%80%ED%86%B5%EA%B3%84%ED%8F%AC%ED%84%B8-%EB%8C%80%EC%9A%A9%EB%9F%89%ED%86%B5%EA%B3%84%EC%9E%90%EB%A3%8C-openAPI-%EC%82%AC%EC%9A%A9

KOSIS(국가통계포털) 대용량통계자료 openAPI 사용

kosis 에서 제공하는건, [통계자료]와 [대용량통계자료] 두가지가 존재한다. 현재로썬, [대용량통계자료] 를 쓸일이 없긴한데 일단 테스트를 진행한다. 자료등록 URL을 생성하고, [URL생성]을 누른다

whiseung.tistory.com

Lambda 함수 생성

lambda 서비스 화면으로 이동하여, 함수를 생성한다.

timeout 시간 변경

기본 타임아웃 설정이 3초라, 10초 정도로 변경해준다.

코드에 작성한 함수 붙여넣기

import json
import io
import boto3
import pandas as pd
from urllib.request import urlopen
from datetime import datetime

ACCESS_KEY = 'access_key'
SECRET_KEY = 'secret_key'

def getApiUploadS3(apiUrl, s3Url, s3Path, tempFileName):
    with urlopen(apiUrl) as url:
        json_file = url.read()
        py_json = json.loads(json_file.decode('utf-8'))

        data = []
        for i, v in enumerate(py_json):
            if i == 0 :
                print(f"Title : {v['TBL_NM']}")

            value = []
            value.append(v['PRD_DE'])
            value.append(v['C1_NM'])
            value.append(v['ITM_NM'])
            value.append(v['DT'])
            value.append(v['UNIT_NM'])

            data.append(value)

    df = pd.DataFrame(data)
    df.columns = ['yyyyymm', 'category', 'item', 'value', 'unit']

    df.to_parquet(s3Url + s3Path + tempFileName
                , engine='pyarrow'
                , index=False
                , storage_options={"key": ACCESS_KEY,
                                   "secret": SECRET_KEY }
                ) 

    return s3Path + tempFileName

def lambda_handler(event, context):
    apiUrl = "<https://kosis.kr/openapi/statisticsData.do?method=getList&apiKey=api_key&format=json&jsonVD=Y&userStatsId=whiseung/115/DT_11523_400/2/1/20221123094346&prdSe=M&newEstPrdCnt=10>"
    bucketName = 'bucket_name'
    s3Path = 'parquet/'
    
    thisTime = datetime.today().strftime('%Y%m%d%H%M%S')
    tempFileName = f'KOSIS_DT_11523_400_{thisTime}.parquet'
    
    resultFileName = getApiUploadS3(apiUrl, f's3://{bucketName}/', s3Path, tempFileName)
    print(f'###  Parquet file uploaded! {resultFileName}')

    return {
        'statusCode': 200,
        'body': json.dumps('Success!')
    }

Deploy 후 Test

테스트를 실행하면, pandas 패키지가 없다는 에러 발생한다.

pandas Layer를 Lambda에 추가

다시 테스트

이번에는 fsspec 이 없다.

내 PC에서 패키지를 다운받아, 올린다.

업로드할때 파일 구성은

Labrary.zip > python > pacakges

c:\\workspace\\api\\lib\\python>dir
 C 드라이브의 볼륨에는 이름이 없습니다.
 볼륨 일련 번호: 1ACD-57FC

 c:\\workspace\\api\\lib\\python 디렉터리

2022-12-01  오후 04:58    <DIR>          .
2022-12-01  오후 04:59    <DIR>          ..
               0개 파일                   0 바이트
               2개 디렉터리  234,113,568,768 바이트 남음

c:\\workspace\\api\\lib\\python>pip3 install fsspec -t . --no-user
Collecting fsspec
  Using cached fsspec-2022.11.0-py3-none-any.whl (139 kB)
Installing collected packages: fsspec
Successfully installed fsspec-2022.11.0
WARNING: You are using pip version 21.1.1; however, version 22.3.1 is available.
You should consider upgrading via the 'C:\\Users\\서휘승\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\\python.exe -m pip install --upgrade pip' command.

c:\\workspace\\api\\lib\\python>dir
 C 드라이브의 볼륨에는 이름이 없습니다.
 볼륨 일련 번호: 1ACD-57FC

 c:\\workspace\\api\\lib\\python 디렉터리

2022-12-02  오전 08:37    <DIR>          .
2022-12-01  오후 04:59    <DIR>          ..
2022-12-02  오전 08:37    <DIR>          fsspec
2022-12-02  오전 08:37    <DIR>          fsspec-2022.11.0.dist-info
               0개 파일                   0 바이트
               4개 디렉터리  234,099,306,496 바이트 남음

c:\\workspace\\api\\lib\\python>

c:\\workspace\\api\\lib\\python>pip3 install s3fs -t . --no-user
Collecting s3fs
  Using cached s3fs-2022.11.0-py3-none-any.whl (27 kB)
Collecting fsspec==2022.11.0
  Using cached fsspec-2022.11.0-py3-none-any.whl (139 kB)
Collecting aiohttp!=4.0.0a0,!=4.0.0a1
  Using cached aiohttp-3.8.3-cp38-cp38-win_amd64.whl (324 kB)
Collecting aiobotocore~=2.4.0
  Using cached aiobotocore-2.4.1-py3-none-any.whl (66 kB)
Collecting wrapt>=1.10.10
  Using cached wrapt-1.14.1-cp38-cp38-win_amd64.whl (35 kB)
Collecting aioitertools>=0.5.1
  Using cached aioitertools-0.11.0-py3-none-any.whl (23 kB)
Collecting botocore<1.27.60,>=1.27.59
  Using cached botocore-1.27.59-py3-none-any.whl (9.1 MB)
Collecting async-timeout<5.0,>=4.0.0a3
  Using cached async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
Collecting aiosignal>=1.1.2
  Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting yarl<2.0,>=1.0
  Using cached yarl-1.8.1-cp38-cp38-win_amd64.whl (56 kB)
Collecting attrs>=17.3.0
  Using cached attrs-22.1.0-py2.py3-none-any.whl (58 kB)
Collecting charset-normalizer<3.0,>=2.0
  Using cached charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
Collecting frozenlist>=1.1.1
  Using cached frozenlist-1.3.3-cp38-cp38-win_amd64.whl (34 kB)
Collecting multidict<7.0,>=4.5
  Using cached multidict-6.0.2-cp38-cp38-win_amd64.whl (28 kB)
Collecting typing_extensions>=4.0
  Using cached typing_extensions-4.4.0-py3-none-any.whl (26 kB)
Collecting python-dateutil<3.0.0,>=2.1
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting urllib3<1.27,>=1.25.4
  Using cached urllib3-1.26.13-py2.py3-none-any.whl (140 kB)
Collecting jmespath<2.0.0,>=0.7.1
  Using cached jmespath-1.0.1-py3-none-any.whl (20 kB)
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting idna>=2.0
  Using cached idna-3.4-py3-none-any.whl (61 kB)
Installing collected packages: six, multidict, idna, frozenlist, yarl, urllib3, typing-extensions, python-dateutil, jmespath, charset-normalizer, attrs, async-timeout, aiosignal, wrapt, botocore, aioitertools, aiohttp, fsspec, aiobotocore, s3fs
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
boto3 1.26.17 requires botocore<1.30.0,>=1.29.17, but you have botocore 1.27.59 which is incompatible.
Successfully installed aiobotocore-2.4.1 aiohttp-3.8.3 aioitertools-0.11.0 aiosignal-1.3.1 async-timeout-4.0.2 attrs-22.1.0 botocore-1.27.59 charset-normalizer-2.1.1 frozenlist-1.3.3 fsspec-2022.11.0 idna-3.4 jmespath-1.0.1 multidict-6.0.2 python-dateutil-2.8.2 s3fs-2022.11.0 six-1.16.0 typing-extensions-4.4.0 urllib3-1.26.13 wrapt-1.14.1 yarl-1.8.1
WARNING: Target directory c:\\workspace\\api\\lib\\python\\fsspec already exists. Specify --upgrade to force replacement.
WARNING: Target directory c:\\workspace\\api\\lib\\python\\fsspec-2022.11.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: You are using pip version 21.1.1; however, version 22.3.1 is available.
You should consider upgrading via the 'C:\\Users\\서휘승\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\\python.exe -m pip install --upgrade pip' command.

c:\\workspace\\api\\lib\\python>

다운받은 라이브러리들을 압축하여 s3에 올린다.

Layer 신규 생성

좌측 메뉴에서 [계층]으로 이동한다.

새로 생성한 Layer를 Lambda에 추가

재실행

Cloud watch 에서 로그보기

s3에서 확인

다음엔 해당 Lambda를 스케쥴링 걸어보겠다.

'AWS' 카테고리의 다른 글

AWS Glue에서 pymssql 을 이용해 MSSQL 데이터를 S3에 저장하기 (0)	2022.12.26
KOSIS openAPI 를 AWS Glue 에서 실행해보기 (1)	2022.12.13
AWS Glue 를 사용하기 위한 IAM 생성 (0)	2022.12.13
API batch 실행을 위해 EventBridge 로 Schedule 생성하여 AWS Lambda 호출 (0)	2022.12.12
python으로 AWS S3 파일 읽고 쓰기 (0)	2022.12.06

현재글KOSIS openAPI 호출을 위한 AWS Lambda 생성

googleappsscript, GoogleForms, GCP, S3, AZURE, spreadsheet, AWS, KAKAO, bigquery, GoogleSheets, GAS, openapi, AppsScript, CognitiveService, gws, Google, FormRecognizer, Python, kosis, Glue,

Today :
Yesterday :

whistory