Imported from TradingAi666/TzFilm-Douyin-Tool (SKILL.md). Install upstream with npx skills add TradingAi666/TzFilm-Douyin-Tool. Copyright stays with the author.

Douyin Creator Center Scraping

The #1 Pitfall: macOS TCC Blocks launchd from Reading ~/Downloads

This is the root cause of 99% of "download timeout" failures. launchd-spawned Python processes cannot read ~/Downloads/, ~/Documents/, or ~/Desktop/ due to macOS TCC (Transparency, Consent, and Control). glob.glob(), os.listdir(), even osascript -e 'do shell script "ls/cp/cat ..."' — ALL silently return empty or zero from launchd context.

The fix: redirect Chrome's download directory to /tmp/. /tmp has zero TCC restrictions. Chrome writes there freely, Python reads there freely.

# One-time setup — persists across Chrome restarts:
defaults write com.google.Chrome DownloadDirectory -string "/tmp/douyin_dl"

After this, download detection is trivially simple:

def fetch_newest_xlsx():
    import glob
    pattern = os.path.join('/tmp/douyin_dl', '作品列表*.xlsx')
    files = glob.glob(pattern)
    if not files:
        return None, None
    newest = max(files, key=os.path.getmtime)
    return newest, os.path.basename(newest)

Detection time: ~2 seconds. Works identically from terminal and launchd.

What does NOT work (field-tested 2026-05-10)

Pitfall #2: Login Session Expiry — Two Failure Modes

Second most common failure after TCC. Douyin creator center sessions expire after a few days. There are TWO distinct failure modes depending on how expired the session is:

Mode A: Recent Expiry → JSON Error File Downloaded

When the session expired recently (hours to ~1 day), clicking "导出数据" still triggers a download — but the server returns a JSON error instead of an xlsx file:

{"BaseResp":{"StatusCode":8,"StatusMessage":"用户未登录"}}

openpyxl then fails with File is not a zip file. The PK header check catches this.

Mode B: Deep Expiry → No File Downloaded At All (2026-05-23)

When the session has been expired for longer (~1+ days), the server returns an error response that Chrome does NOT interpret as a file download. No file ever appears in the download directory. The wait_for_download PK check never fires because glob returns [] every iteration. The scraper logs only:

点击导出: clicked:3
  等待下载... 10s
  等待下载... 20s
  ...
❌ 等待下载超时

Crucially, there are NO ⚠️ 非xlsx文件 or ❌ 登录态过期 log lines — the file was never downloaded, so the PK validation code path is never reached.

Degradation: Garfish App Stops Initializing

After too many reload attempts with an expired session, the page degrades further. The Garfish micro-frontend stops initializing entirely:

• document.body.innerText shows only sidebar navigation (~66 chars): "高清发布 首页 活动管理..."
• Garfish container has __garfishmockhtml__ attribute — the placeholder HTML, not the live app
• "加载中，请稍候..." text may appear
• document.querySelectorAll('button') returns only 1 button ("高清发布")
• "投稿列表" tab and "导出数据" button are never found

This is the same root cause as Mode B (expired session) but the symptoms are different — it looks like the page failed to load, not that the download failed.

Detection: Is It Session Expiry or Something Else?

⚠️ Health Check Limitation

The pre-flight health check (health_check()) validates AppleScript JS access and tab existence — it does NOT validate the Douyin login session. A health check returning js_ok means the AppleScript pipeline works, but the user may still be logged out.

Recovery: User must re-login to creator.douyin.com in Chrome. No script changes needed. After re-login, verify with a manual scrape run: python3 ~/.hermes/scripts/douyin_hourly.py

Pitfall #3: Page Stuck on Home → Tabs/Buttons Not Found

After login, Chrome may land on the creator home page (/creator-micro/home). The Garfish data-center app never loads there.

Fix: force-navigate URL in reload_page() — never just reload t:

set URL of t to "https://creator.douyin.com/creator-micro/data-center/content"
delay 3
reload t

Pre-flight Health Check (MUST run before scrape)

Before attempting any scrape, validate the AppleScript pipeline with a lightweight JS execution test. This catches 3 common silent failures before wasting time on a full page reload + Garfish wait cycle:

def health_check():
    """Returns (ok, reason). Runs in ~2 seconds if healthy."""
    test_js = "/tmp/dy_health_check.js"
    with open(test_js, "w") as f:
        f.write("(function(){ return 'js_ok'; })()")
    
    script = f'''
    tell application "Google Chrome"
        repeat with w in windows
            repeat with t in tabs of w
                if (URL of t) contains "creator.douyin.com" then
                    set js to read POSIX file "{test_js}"
                    return execute t javascript js
                end if
            end repeat
        end repeat
        return "no_tab"
    end tell
    '''
    try:
        result = subprocess.check_output(['osascript', '-e', script], timeout=15).decode()
    except subprocess.TimeoutExpired:
        return False, "AppleScript 执行超时"
    except Exception as e:
        if "Allow JavaScript" in str(e) or "JavaScript 的功能已关闭" in str(e):
            return False, "🔧 AppleScript JS 已关闭 → Chrome 菜单勾上"
        return False, f"异常: {str(e)[:100]}"
    
    if result == "no_tab":
        return False, "📄 未找到 creator.douyin.com 标签页"
    if "js_ok" not in result:
        return False, f"⚠️ JS 返回异常: {result[:100]}"
    
    return True, "ok"

Call this at the top of main(). If it fails, send the reason string directly to Telegram — the user gets an actionable error message instead of a generic timeout.

Why this matters: AppleScript JS silently resets (Chrome updates, re-logins, desktop switches). Without this check, the scrape silently fails after 30s of polling for buttons that never appear. With it, failure is detected in ~2s with a clear fix instruction.

Troubleshooting: "no_tab" — Diagnostic to List All Douyin Tabs

When the health check returns "no_tab" but the user says Chrome is open, run this diagnostic to find ALL douyin-related tabs across every window. Use execute_code (not terminal) because the AppleScript & string concatenation operator is misinterpreted as shell backgrounding by the terminal tool:

import subprocess

script = '''
tell application "Google Chrome"
    set output to ""
    repeat with w in windows
        repeat with t in tabs of w
            if URL of t contains "douyin" then
                set output to output & (URL of t) & " | win_id:" & (id of w) & linefeed
            end if
        end repeat
    end repeat
    if output is "" then
        return "NO_DOUYIN_TABS"
    end if
    return output
end tell
'''

result = subprocess.check_output(['osascript', '-e', script], timeout=10).decode().strip()
print(result)

This reveals:

• Whether ANY douyin tab exists at all
• Which sub-page each tab is on (/content/manage, /data-center/content, /home, etc.)
• Which Chrome window each tab lives in (useful for Spaces/Desktop debugging)

Common findings:

Prerequisites

• macOS with Google Chrome open and logged into creator.douyin.com
• AppleScript JS execution enabled in Chrome: View → Developer → Allow JavaScript from Apple Events ⚠️ This setting resets frequently (Chrome updates, re-logins, desktop switches).
• Chrome download directory: defaults write com.google.Chrome DownloadDirectory -string "/tmp/douyin_dl"
• ⚠️ macOS desktop aware: Chrome must be on the current active desktop/Space. AppleScript silently fails when Chrome is on a different Space.
• ⚠️ "Allow JavaScript from Apple Events" can silently reset. Symptom: execution error: JavaScript execution via AppleScript is turned off.
• Chrome download directory set to /tmp/douyin_dl/: defaults write com.google.Chrome DownloadDirectory -string "/tmp/douyin_dl"
• Chrome must be on the active macOS desktop — switching spaces can break AppleScript tab operations
• ⚠️ macOS desktop / Space aware: Chrome must be on the current active macOS desktop (Space). AppleScript interactions silently fail or produce stale results when Chrome lives on a different Space. If the scraper suddenly stops finding tabs/buttons after working fine before, check whether Chrome got moved to another desktop.
• ⚠️ "Allow JavaScript from Apple Events" can silently reset. Chrome updates and re-logins can uncheck this setting. Symptom: execution error: "Google Chrome" has encountered an error: JavaScript execution via AppleScript is turned off. Fix: re-check the menu item.

Architecture

The Douyin creator center uses Garfish (ByteDance micro-frontend framework):

• Content lives inside #micro > #garfish_app_for_* (ID suffix changes on reload)
• Use document.querySelectorAll on document (not garfish container)
• document.body.innerText works after full page load + tab switch

Step-by-Step Flow

1. Force-navigate to data center + reload — reload_page() MUST set the URL to https://creator.douyin.com/creator-micro/data-center/content before reloading, NOT just call reload t. Douyin's login redirect often lands on the home page (/creator-micro/home), and reloading there keeps you on home — the Garfish data-center app never loads. The AppleScript MUST:

   set URL of t to "https://creator.douyin.com/creator-micro/data-center/content"
   delay 3
   reload t

   Also: activate + set active tab index + set index of w to 1 are required.
2. Wait 15s for Garfish app to initialize
3. Switch to 投稿列表 tab — charCode match: 25237,31295,21015,34920
4. Click 刷新数据 — charCode: 21047,26032,25968,25454. Wait 4s.
5. Poll for 导出数据 button — up to 15 attempts, 2s each
6. Click first 导出数据 button — charCode: 23548,20986,25968,25454
7. Wait for 作品列表*.xlsx in /tmp/douyin_dl/ — poll with glob every 1s, validate PK header, timeout 40s
8. Parse with openpyxl, delete file after

Code vs SKILL.md Consistency (2026-05-22 Fix)

The script douyin_hourly.py was out of sync with this SKILL.md on two critical points:

1. reload_page() did reload t without setting URL → now sets URL to /data-center/content
2. wait_for_download() called move_latest_xlsx_via_applescript() looking in ~/Downloads → now uses direct glob on /tmp/douyin_dl

Both fixes are in the open-source repo. The SKILL.md was always correct — the code just wasn't following it.

Critical: Tab MUST Be Foreground

Chrome blocks downloads triggered by JS click() on background tabs. Before any interaction, the AppleScript MUST:

tell application "Google Chrome"
    activate
    set active tab index of w to tabIndex
    set index of w to 1
end tell

All three (activate + active tab index + index of w to 1) are required.

AppleScript: No Unicode Escapes

Never use \uXXXX in JS via AppleScript — they fail silently. Use charCodeAt():

// 导出数据: 23548,20986,25968,25454
// 投稿列表: 25237,31295,21015,34920
// 刷新数据: 21047,26032,25968,25454
if(t.charCodeAt(0)===23548 && t.charCodeAt(1)===20986 ...) { ... }

Always write JS to /tmp/*.js file, then read POSIX file in AppleScript.

Excel Schema

作品名称 | 发布时间 | 体裁 | 审核状态 | 播放量 | 完播率 | 5s完播率 |
封面点击率 | 2s跳出率 | 平均播放时长 | 点赞量 | 分享量 | 评论量 | 收藏量 |
主页访问量 | 粉丝增量

~99 rows per export. Percentages as decimals — multiply by 100.

Database

CREATE TABLE video_stats (
    timestamp DATETIME, title TEXT, publish_date TEXT,
    plays INTEGER, avg_duration_sec INTEGER, ctr REAL,
    finish_rate REAL,
    likes INTEGER, comments INTEGER, shares INTEGER,
    favorites INTEGER, danmaku INTEGER, status TEXT
)

• ctr = 5s完播率 (%), stored from xlsx column 「5s完播率」
• finish_rate = 完播率/播放占比 (%), stored from xlsx column 「完播率」 (2026-05-23 新增)
• avg_duration_sec = average watch duration in seconds
• status = 审核状态 («公开», «自见», «私密», «未通过», etc.) — added 2026-05-23. Must use ALTER TABLE ADD COLUMN + explicit column names in INSERT to avoid column ordering bugs (see Pitfall #7). Filter status NOT IN ('自见', '私密', '未通过', '审核中', '已删除') when syncing to Feishu — feishu_sync.py applies this filter in the SELECT query.

publish_date has 3 formats mixed (short ISO / full ISO / Chinese). Always match on title only, never publish_date.

Pitfall #8: status Field Not Stored → All Videos Synced (2026-05-23)

The scraper parsed «审核状态» from xlsx but didn't store it in the DB (line 348 reads it, line 388 didn't include it in INSERT). feishu_sync.py synced ALL 102 videos including 私密/自见 ones. Fix: ① ALTER TABLE video_stats ADD COLUMN status TEXT ② INSERT includes status with explicit column names ③ feishu_sync.py filters WHERE status NOT IN ('自见', '私密', '未通过', '审核中', '已删除'). Cleanup: after fix, manually delete stale Feishu records that match private video titles — sync is upsert-only, never deletes.

launchd Deployment

<!-- ~/Library/LaunchAgents/com.hermes.douyin-tracker.plist -->
<key>StartInterval</key><integer>3600</integer>
<key>EnvironmentVariables</key>
<dict>
    <key>HOME</key><string>/Users/YOUR_USER</string>
    <key>PATH</key><string>/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
</dict>

Reload: launchctl unload/load ~/Library/LaunchAgents/...plist Trigger: launchctl kickstart -k gui/$(id -u)/com.hermes.douyin-tracker

Pitfall #4: Douyin UI Redesign → Export Button Gone (2026-05-22)

Douyin rolled out a new creator-center UI on 2026-05-22. The old /data-center/content page (with "投稿列表" tab and "导出数据" button) was replaced by /content/manage (with "全部作品/已发布/审核中/未通过" tabs and no export button anywhere).

The fix: force-navigate to the OLD URL. The old Garfish micro-frontend at /data-center/content is still served and still has the export button. reload_page() MUST set the URL before reloading — never just reload t:

set URL of t to "https://creator.douyin.com/creator-micro/data-center/content"
delay 3
reload t

Pitfall #5: Download Detection — Don't Move from ~/Downloads

When Chrome's DownloadDirectory is set to /tmp/douyin_dl (as recommended in Pitfall #1), the move_latest_xlsx_via_applescript() approach fails because files are never in ~/Downloads. Poll /tmp/douyin_dl directly with glob instead — it's faster (~2s vs 40s+ timeout) and more reliable:

def wait_for_download(before_ts, timeout=40):
    import glob
    for i in range(timeout):
        time.sleep(1)
        pattern = os.path.join(DOWNLOADS_DIR, '作品列表*.xlsx')
        files = sorted(glob.glob(pattern), key=os.path.getmtime, reverse=True)
        for f in files:
            if os.path.getmtime(f) > before_ts:
                with open(f, 'rb') as fh:
                    if fh.read(2) != b'PK':  # validate real xlsx
                        ...  # login expiry check
                        continue
                return f
    return None

Pitfall #6: Stale launchd Process → Stale Data (2026-05-22)

If the scraper's last data update was hours/days ago but launchctl list shows it running (PID present, exit code 0), a zombie process may be stuck. The process survived a Chrome/network hiccup but never exited.

Detection:

# Check for multiple instances:
ps aux | grep douyin_hourly
# If older PID still running alongside new ones, kill it

Fix: Kill the stale PID, then manually trigger a run or wait for next schedule:

kill <stale_pid>
launchctl kickstart -k gui/$(id -u)/com.hermes.douyin-tracker

Symptoms:

• douyin_hourly.log shows repeated "not found" for export button (10+ retries)
• Latest data timestamp is hours old despite job running "successfully"
• Multiple douyin_hourly.py processes in ps aux

Pitfall #7: ALTER TABLE Column Ordering vs INSERT Without Column Names (2026-05-23)

When adding a new column (e.g. finish_rate) to an existing table via ALTER TABLE, the column is added at the end of the table — regardless of where it appears in CREATE TABLE IF NOT EXISTS. If the INSERT statement uses positional VALUES without explicit column names, data goes into the wrong columns.

Example of the bug:

# Schema after ALTER TABLE: timestamp, title, ..., danmaku, finish_rate
# But CREATE TABLE IF NOT EXISTS has finish_rate before likes
# Since table already exists, CREATE TABLE is a no-op — schema unchanged
cursor.execute(
    "INSERT INTO video_stats VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
    (now, title, pub_date, plays, duration, ctr5s, finish,  # finish → position 6
     likes, comments, shares, favorites, 0)                 # but position 6 is still 'likes'
)
# Result: finish_rate column gets 0 (the last placeholder),
#         finish value goes to likes column

Fix: Always use explicit column names in INSERT after schema migration:

cursor.execute(
    "INSERT INTO video_stats (timestamp, title, publish_date, plays, "
    "avg_duration_sec, ctr, finish_rate, likes, comments, shares, "
    "favorites, danmaku) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
    (...)
)

Detection: All rows show finish_rate = 0 or wrong values in adjacent columns.

Open-Source Repo (2026-05-22)

The complete tool suite is now public at TradingAi666/TzFilmdouyintool (formerly douyin-creator-scraping, renamed 2026-05-22):

• douyin_hourly.py — hourly data scraping (AppleScript)
• douyin_new_video_tracker.py — video performance prediction
• prediction_query.py — query trending data
• auto_reply.py — Python wrapper for comment auto-reply
• auto-reply/ — Node.js + Playwright browser automation for comment export/reply

Pitfall #7: ALTER TABLE Column Ordering vs INSERT Without Column Names (2026-05-23)

When adding a new column (e.g. finish_rate) to an existing table via ALTER TABLE, the column is added at the end — regardless of CREATE TABLE IF NOT EXISTS ordering. If INSERT uses positional VALUES without explicit column names, data goes into wrong columns.

Fix: Always use explicit column names after schema migration:

cursor.execute(
    "INSERT INTO video_stats (timestamp, title, publish_date, plays, "
    "avg_duration_sec, ctr, finish_rate, likes, comments, shares, "
    "favorites, danmaku) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
    (...)
)

Detection: finish_rate = 0 for all rows despite debug showing correct values.

Pitfall #9: Stale fcntl Lock Blocks All Scrapes (2026-05-24)

Root cause A — flock() leaks through subprocess file descriptors: douyin_hourly.py uses fcntl.flock() to prevent concurrent Chrome access. After acquiring the lock, it calls subprocess.run(['python3', 'feishu_sync.py']). Without close_fds=True, the child process inherits ALL open file descriptors, including lock_fd. flock() locks are attached to the open file description (kernel-level), not the file descriptor. Even when the parent's finally block closes lock_fd, the child still holds a reference → lock never released until child exits (up to 300s timeout).

Root cause B — stale lock from crash: If Python process crashes or is killed, 0-byte lock file persists on disk.

Symptoms:

• Log shows: ⚠️ 已有抓取进程在运行，跳过本次 every tick, for hours
• ps aux | grep douyin returns nothing — NO process running
• Lock file is 0 bytes but flock() returns EAGAIN

Fix:

# In douyin_hourly.py AND douyin_new_video_tracker.py (3 locations total):
_sp.run(['python3', 'feishu_sync.py'],
        timeout=300, capture_output=True, close_fds=True)  # ← close_fds=True

Recovery (stale lock):

rm -f ~/.hermes/.douyin_scrape.lock

Pitfall #10: scrape_total_followers — Garfish Isolation + AppleScript Fix (2026-05-24)

The scrape_total_followers() function must navigate to /creator-micro/home to extract fans because the Garfish micro-frontend at /data-center/content does NOT contain the sidebar in document.body.innerText. The sidebar (with "关注 N 粉丝 N.N万") lives in the parent frame outside Garfish.

Fix: AppleScript navigates to home page with proper exit repeat, then runs JS:

tell application "Google Chrome"
    activate
    repeat with w in windows
        if found then exit repeat
        repeat with t in tabs of w
            if (URL of t) contains "creator.douyin.com" then
                set URL of t to "https://creator.douyin.com/creator-micro/home"
                delay 6
                set js to read POSIX file "/tmp/dy_follower_js_v2.js"
                return execute t javascript js
            end if
        end repeat
    end repeat
end tell

JS pattern: body.match(/\u5173\u6ce8\s*[\d,]+\s*\u7c89\u4e1d\s*([\d.]+)\s*\u4e07/) — matches "关注 N 粉丝 N.N万".

Timing: Fan extraction runs at step 10 (after data-center export + Telegram report), when navigation to home won't break anything. close_fds=True on subprocess prevents lock leak.

Pitfall #11: today_new_fans Always 0 — Page Has No Daily New Fans Field (2026-05-24)

The JS scraping searched document.body.innerText for "新增" (charCode 26032, 22686) to find today's new follower count. The Douyin creator center home page does NOT contain a single-day "今日新增粉丝" metric. The page only shows:

• Total fans (e.g., 4.89万) — in sidebar
• 7-day net new ("净增粉丝 1.18万") — in data center section
• Per-followed-account "昨日 +N" — in followed accounts section

Symptom: account_stats.today_new_fans is always 0, even though total_fans is increasing (48100→48900).

Fix: Compute today_new from DB deltas in save_account_stats():

def save_account_stats(data):
    current_fans = data['fans']
    today_start = datetime.now().strftime("%Y-%m-%d") + " 00:00:00"
    
    # Baseline: first entry of today's total fans
    first_today = cur.execute(
        "SELECT total_fans FROM account_stats WHERE timestamp >= ? AND total_fans > 0 ORDER BY timestamp ASC LIMIT 1",
        (today_start,)
    ).fetchone()
    
    if first_today and first_today[0] > 0:
        today_new = current_fans - first_today[0]
    else:
        # Fallback: difference from yesterday's last entry
        yesterday_end = datetime.now().strftime("%Y-%m-%d") + " 00:00:00"
        last_yesterday = cur.execute(
            "SELECT total_fans FROM account_stats WHERE timestamp < ? AND total_fans > 0 ORDER BY timestamp DESC LIMIT 1",
            (yesterday_end,)
        ).fetchone()
        today_new = current_fans - (last_yesterday[0] if last_yesterday else current_fans)
    
    cur.execute("INSERT INTO account_stats VALUES (?, ?, ?)",
                (now, current_fans, today_new))

Effect: At 12:07, log("💾 粉丝数据已入库: 48900 (+400)") — today_new correctly shows 400 new followers (48900 - 48500 = 400, where 48500 was the 06:06 first-of-day entry).

The scrape_total_followers() function originally used AppleScript to navigate to creator-micro/home, looping through ALL Chrome windows/tabs without exit repeat. With many open tabs, the delay 5 per matching tab caused the 20s timeout.

Fix: Remove AppleScript navigation entirely. The creator center sidebar shows "粉丝 N.N万" on every page — no need to navigate to home. Extract directly from the current page's document.body.innerText.

Before (48 lines):

def scrape_total_followers():
    # AppleScript: navigate to home (20s timeout!)
    script = '''tell application "Google Chrome"
        activate
        repeat with w in windows       # ← no exit repeat!
            repeat with t in tabs of w  # ← loops ALL tabs
                if URL contains "creator.douyin.com" then
                    set URL to "creator-micro/home"
                    delay 5              # ← 5s delay × N tabs → timeout
                end if
            end repeat
        end repeat
    end tell'''
    subprocess.check_output(['osascript', '-e', script], timeout=20)
    time.sleep(3)
    # ... then JS extraction

After (~30 lines):

def scrape_total_followers():
    """从当前页面侧边栏提取粉丝总量（无需导航）"""
    js = r"""(function(){
        var result = {fans: 0};
        var body = document.body.innerText.replace(/\n/g, ' ');
        // Find 粉丝 by charCode (see Pitfall #7)
        ...
        return JSON.stringify(result);
    })()"""
    # Write JS to file → AppleScript execute → no navigation needed
    raw = run_js_file('/tmp/dy_follower_js.js', timeout=15)

The scraper chains a Feishu sync as step 10 after Telegram report — see douyin-automation/references/feishu-sync.md.

Existing Infrastructure

• Script: ~/.hermes/scripts/douyin_hourly.py
• Log: ~/.hermes/logs/douyin_hourly.log
• DB: ~/.hermes/douyin_stats.db
• Download dir: /tmp/douyin_dl/