2026-08-18 04:23:14 [scrapy.utils.log] INFO: Scrapy 2.10.0 started (bot: sitebunny)
2026-08-18 04:23:14 [scrapy.utils.log] INFO: Versions: lxml 4.9.3.0, libxml2 2.10.3, cssselect 1.2.0, parsel 1.8.1, w3lib 2.1.2, Twisted 22.10.0, Python 3.11.13 (main, Jul 22 2025, 01:43:09) [GCC 10.2.1 20210110], pyOpenSSL 23.2.0 (OpenSSL 3.1.2 1 Aug 2023), cryptography 41.0.3, Platform Linux-5.15.0-141-generic-x86_64-with-glibc2.31
2026-08-18 04:23:14 [scrapy.addons] INFO: Enabled addons:
[]
2026-08-18 04:23:14 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'sitebunny',
 'CLOSESPIDER_PAGECOUNT': '2001',
 'CONCURRENT_REQUESTS': '4',
 'COOKIES_ENABLED': False,
 'DEPTH_LIMIT': '10',
 'DEPTH_PRIORITY': 1,
 'DOWNLOAD_DELAY': 1,
 'FEED_EXPORT_ENCODING': 'utf-8',
 'LOG_FILE': '/data/webcrawl/scrapyd.dev/logs/sitebunny/sb-spider-v1/863e89269abc11f1ba910242ac130002.log',
 'LOG_LEVEL': 'INFO',
 'NEWSPIDER_MODULE': 'sitebunny.spiders',
 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7',
 'ROBOTSTXT_OBEY': 'False',
 'SCHEDULER_DISK_QUEUE': 'scrapy.squeues.PickleFifoDiskQueue',
 'SCHEDULER_MEMORY_QUEUE': 'scrapy.squeues.FifoMemoryQueue',
 'SPIDER_MODULES': ['sitebunny.spiders'],
 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'}
2026-08-18 04:23:14 [scrapy.extensions.telnet] INFO: Telnet Password: d20db98d878f2a11
2026-08-18 04:23:14 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.closespider.CloseSpider',
 'scrapy.extensions.logstats.LogStats']
2026-08-18 04:23:14 [sb-spider-v1] INFO: {'allow': 'https?://www.42maru.ai', 'deny': 'signin|sign-in|Signin|SignIn|signIn|login|log-in|Login|LogIn|logIn|signup|sign-up|Signup|SignUp|signUp|findid|find-id|FindId|FindID|findId|findID|findpw|find-pw|FindPw|FindPW|findPw|findPW|findpassword|find-password|FindPassword|findPassword|findaccount|find-account|FindAccount|findAccount|/idpw|forgot-password|change-password|mode=download|downloadFile|/file/download|download.php|/download|/bbs/down.php|attachFile/download|getVcard.|dispBoardTagList|.(jpg|pdf|hwp|xls|xlsx|doc|docx|zip)$'}
2026-08-18 04:23:14 [sb-spider-v1] INFO: {'link_extractor': <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor object at 0x7f3941bf3b10>, 'callback': 'parse_item', 'follow': True, 'process_request': 'set_playwright', 'errback': 'errback'}
2026-08-18 04:23:14 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2026-08-18 04:23:14 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2026-08-18 04:23:14 [scrapy.middleware] INFO: Enabled item pipelines:
['sitebunny.pipelines.SitebunnyPipeline']
2026-08-18 04:23:14 [scrapy.core.engine] INFO: Spider opened
2026-08-18 04:23:14 [scrapy.extensions.closespider] INFO: Spider will stop when no items are produced after 300 seconds.
2026-08-18 04:23:14 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2026-08-18 04:23:14 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2026-08-18 04:23:14 [scrapy-playwright] INFO: Starting download handler
2026-08-18 04:23:14 [scrapy-playwright] INFO: Starting download handler
2026-08-18 04:23:19 [sb-spider-v1] INFO: Message sent successfully to Slack!
2026-08-18 04:23:19 [scrapy-playwright] INFO: Launching browser chromium
2026-08-18 04:23:19 [scrapy-playwright] INFO: Browser chromium launched
2026-08-18 04:23:27 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/ None None
2026-08-18 04:23:39 [root] INFO: 신규 페이지 저장
2026-08-18 04:23:46 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/ 1 1
2026-08-18 04:23:46 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/contact/ 1 1
2026-08-18 04:23:55 [root] INFO: 신규 페이지 저장
2026-08-18 04:23:56 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/ 2 2
2026-08-18 04:23:56 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/kr/ 2 2
2026-08-18 04:24:06 [root] INFO: 신규 페이지 저장
2026-08-18 04:24:08 [root] INFO: 기수집. 미업데이트. update last_crawled_at
2026-08-18 04:24:13 [root] INFO: 기수집. 미업데이트. update last_crawled_at
2026-08-18 04:24:14 [scrapy.extensions.logstats] INFO: Crawled 6 pages (at 6 pages/min), scraped 5 items (at 5 items/min)
2026-08-18 04:24:14 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/blog/ 5 5
2026-08-18 04:24:23 [root] INFO: 신규 페이지 저장
2026-08-18 04:24:23 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/about-us/ 6 6
2026-08-18 04:24:23 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/about-us/partners/ 6 6
2026-08-18 04:24:32 [root] INFO: 신규 페이지 저장
2026-08-18 04:24:41 [root] INFO: 신규 페이지 저장
2026-08-18 04:24:46 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/about-us/our-team/ 8 8
2026-08-18 04:24:57 [root] INFO: 신규 페이지 저장
2026-08-18 04:24:58 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/about-us/investors/ 9 9
2026-08-18 04:24:58 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/solutions/ 9 9
2026-08-18 04:24:58 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/about-us/history/ 9 9
2026-08-18 04:25:08 [root] INFO: 신규 페이지 저장
2026-08-18 04:25:17 [root] INFO: 신규 페이지 저장
2026-08-18 04:25:17 [scrapy.extensions.logstats] INFO: Crawled 12 pages (at 6 pages/min), scraped 11 items (at 6 items/min)
2026-08-18 04:25:27 [root] INFO: 신규 페이지 저장
2026-08-18 04:25:34 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/solutions/ai-large-enterprise/ 12 12
2026-08-18 04:25:44 [root] INFO: 신규 페이지 저장
2026-08-18 04:25:44 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/solutions/ai-smart-devices/ 13 13
2026-08-18 04:25:44 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/solutions/ai-based-chatbot/ 13 13
2026-08-18 04:25:44 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/cases/ 13 13
2026-08-18 04:25:54 [root] INFO: 신규 페이지 저장
2026-08-18 04:26:04 [root] INFO: 신규 페이지 저장
2026-08-18 04:26:13 [root] INFO: 신규 페이지 저장
2026-08-18 04:26:14 [scrapy.extensions.logstats] INFO: Crawled 16 pages (at 4 pages/min), scraped 16 items (at 5 items/min)
2026-08-18 04:26:19 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/technology/ 16 16
2026-08-18 04:26:28 [root] INFO: 신규 페이지 저장
2026-08-18 04:26:28 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/cases/ai-document-search-and-comparison/ 17 17
2026-08-18 04:26:29 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/cases/customer-ai-based-chatbot/ 17 17
2026-08-18 04:26:29 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/cases/standard-specification-comparison/ 17 17
2026-08-18 04:26:37 [root] INFO: 신규 페이지 저장
2026-08-18 04:26:46 [root] INFO: 신규 페이지 저장
2026-08-18 04:26:55 [root] INFO: 신규 페이지 저장
2026-08-18 04:27:02 [sb-spider-v1] INFO: request callback: 200, request_sub_id: 0, processed_item_count: 20
2026-08-18 04:27:02 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/cases/contract-comprehension-and-comparison/ 20 20
2026-08-18 04:27:12 [root] INFO: 신규 페이지 저장
2026-08-18 04:27:13 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/blog/42maru-has-thesis-adopted-acl/ 21 21
2026-08-18 04:27:13 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/cases/ 21 21
2026-08-18 04:27:13 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/blog/ 21 21
2026-08-18 04:27:22 [root] INFO: 신규 페이지 저장
2026-08-18 04:27:22 [scrapy.extensions.logstats] INFO: Crawled 24 pages (at 8 pages/min), scraped 22 items (at 6 items/min)
2026-08-18 04:27:24 [root] INFO: 기수집. 미업데이트. update last_crawled_at
2026-08-18 04:27:26 [root] INFO: 기수집. 미업데이트. update last_crawled_at
2026-08-18 04:27:32 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/tags/?tag=Research 24 24
2026-08-18 04:27:43 [root] INFO: 신규 페이지 저장
2026-08-18 04:27:43 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/tags/?tag=tv 25 25
2026-08-18 04:27:43 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/tags/?tag=program 25 25
2026-08-18 04:27:44 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/blog/kdb-bank-selects-42maru-innovative-startup/ 25 25
2026-08-18 04:27:54 [root] INFO: 신규 페이지 저장
2026-08-18 04:28:04 [root] INFO: 신규 페이지 저장
2026-08-18 04:28:15 [root] INFO: 신규 페이지 저장
2026-08-18 04:28:15 [scrapy.extensions.logstats] INFO: Crawled 28 pages (at 4 pages/min), scraped 28 items (at 6 items/min)
2026-08-18 04:28:20 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/blog/recruiting-kbs-2020-job-fair/ 28 28
2026-08-18 04:28:29 [root] INFO: 신규 페이지 저장
2026-08-18 04:28:30 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/tags/?tag=hr 29 29
2026-08-18 04:28:30 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/about-us/careers/ 29 29
2026-08-18 04:28:30 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/en/legal/ 29 29
2026-08-18 04:28:39 [root] INFO: 신규 페이지 저장
2026-08-18 04:28:39 [sb-spider-v1] INFO: 파싱 제외. response status 404
2026-08-18 04:28:47 [root] INFO: 신규 페이지 저장
2026-08-18 04:28:53 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/kr/contact/ 31 31
2026-08-18 04:29:02 [root] INFO: 신규 페이지 저장
2026-08-18 04:29:02 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/kr/Notice/ 32 32
2026-08-18 04:29:02 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/kr/ 32 32
2026-08-18 04:29:02 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/kr/blog/ 32 32
2026-08-18 04:29:12 [root] INFO: 신규 페이지 저장
2026-08-18 04:29:15 [root] INFO: 기수집. 미업데이트. update last_crawled_at
2026-08-18 04:29:15 [scrapy.extensions.logstats] INFO: Crawled 36 pages (at 8 pages/min), scraped 34 items (at 6 items/min)
2026-08-18 04:29:25 [root] INFO: 신규 페이지 저장
2026-08-18 04:29:31 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/kr/%EB%8B%B9%EC%82%AC-%EC%82%AC%EC%B9%AD-%EC%A3%BC%EC%9D%98-%EC%95%88%EB%82%B4/ 35 35
2026-08-18 04:29:40 [root] INFO: 신규 페이지 저장
2026-08-18 04:29:40 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/kr/%EC%9E%85%EC%B0%B0-%EA%B3%B5%EA%B3%A0-ai-%EB%AA%A8%EB%8D%B8-%ED%95%99%EC%8A%B5%EC%9D%84-%EC%9C%84%ED%95%9C-gpu%EC%84%9C%EB%B2%84-%EA%B5%AC%EB%A7%A4/ 36 36
2026-08-18 04:29:40 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/kr/blog/42MaruWIS/ 36 36
2026-08-18 04:29:40 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/kr/blog/ 36 36
2026-08-18 04:29:56 [root] INFO: 신규 페이지 저장
2026-08-18 04:30:10 [root] INFO: 신규 페이지 저장
2026-08-18 04:30:13 [root] INFO: 기수집. 미업데이트. update last_crawled_at
2026-08-18 04:30:14 [scrapy.extensions.logstats] INFO: Crawled 40 pages (at 4 pages/min), scraped 39 items (at 5 items/min)
2026-08-18 04:30:19 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/kr/tags/?tag=CHAT42 39 39
2026-08-18 04:30:31 [root] INFO: 신규 페이지 저장
2026-08-18 04:30:33 [sb-spider-v1] INFO: request callback: 200, request_sub_id: 1, processed_item_count: 40
2026-08-18 04:30:33 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/kr/tags/?tag=WIS2022 40 40
2026-08-18 04:30:34 [sb-spider-v1] INFO: request callback: 200, request_sub_id: 2, processed_item_count: 40
2026-08-18 04:30:34 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/kr/tags/?tag=ANSWERINGAI 40 40
2026-08-18 04:30:35 [sb-spider-v1] INFO: request callback: 200, request_sub_id: 3, processed_item_count: 40
2026-08-18 04:30:35 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/kr/tags/?tag=WorldITShow 40 40
2026-08-18 04:30:47 [root] INFO: 신규 페이지 저장
2026-08-18 04:30:56 [root] INFO: 신규 페이지 저장
2026-08-18 04:31:05 [root] INFO: 신규 페이지 저장
2026-08-18 04:31:11 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/kr/tags/?tag=WIS 43 43
2026-08-18 04:31:20 [root] INFO: 신규 페이지 저장
2026-08-18 04:31:20 [scrapy.extensions.logstats] INFO: Crawled 45 pages (at 5 pages/min), scraped 44 items (at 5 items/min)
2026-08-18 04:31:21 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/kr/tags/?tag=%EC%9B%94%EB%93%9CIT%EC%87%BC 44 44
2026-08-18 04:31:21 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/kr/tags/?tag=AI%ED%94%8C%EB%9E%AB%ED%8F%BC 44 44
2026-08-18 04:31:21 [sb-spider-v1] INFO: Parse Item function called on https://www.42maru.ai/kr/tags/?tag=%ED%8F%AC%ED%8B%B0%ED%88%AC%EB%A7%88%EB%A3%A8 44 44
2026-08-18 04:31:33 [root] INFO: 신규 페이지 저장
2026-08-18 04:31:46 [root] INFO: 신규 페이지 저장
2026-08-18 04:31:56 [root] INFO: 신규 페이지 저장