Command Palette

Search for a command to run...

Hectal
PHASE 4Beginner ~21 min· topic 6 of 7

Topic 4.6

Data from Cloud Storage (S3 or GCS)

In one line

Later the data will live in cloud storage instead of your project folder. Only one line of config changes, but the browser then enforces CORS, caching and size rules you must understand. Keys and secrets never go in frontend code.

0/7 · 0%

Think of it like this

Until now the dashboard read its files from a cupboard in its own house. Now they will sit in a storage unit across town. The dashboard only needs the new address, but the storage unit has its own door rules: who may come in, how long a visitor may keep a copy, and what a visitor may carry out.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

bucket
A named container for files in S3 or GCS. Each file (an object) has a key such as lumen/daily.csv.
origin
Scheme + host + port of a page or server, such as http://localhost:5173.
CORS
Cross-Origin Resource Sharing: the rules by which a server tells browsers which other origins may read its answers.
Cache-Control
A header that says how long a copy may be reused, such as max-age=3600.
gzip
A compression format. Text shrinks to roughly a third.
presigned URL
A temporary signed link to one private file. It needs no key in the browser and stops working at its expiry.

Step by step

01Change one line to move the data

config.js holds export const DATA_URL = "/data/lumen/";. Pointing it at a bucket changes only this value, because load.js builds every URL from it. The first example shows the helper and what each base does.

Change one line to move the datadiagram
Rendering diagram…

02Allow your page's origin (bucket setting)

The bucket owner adds this rule. It is not code in your dashboard. The origin is a placeholder: use your own.

S3 bucket CORS configuration (placeholder origins)whole filejson
[
  {
    "AllowedOrigins": ["https://YOUR-DASHBOARD.example", "http://localhost:5173"],
    "AllowedMethods": ["GET"],
    "AllowedHeaders": ["*"],
    "MaxAgeSeconds": 3600
  }
]

03The same rule in Google Cloud Storage

Save this as cors.json and apply it in Cloud Shell with gcloud storage buckets update gs://YOUR-BUCKET --cors-file=cors.json. Then check a file nobody has fetched yet: the answer must carry Access-Control-Allow-Origin.

cors.json for a GCS bucket (placeholder origins)whole filejson
[
  {
    "origin": ["https://YOUR-DASHBOARD.example", "http://localhost:5173"],
    "method": ["GET"],
    "responseHeader": ["Content-Type"],
    "maxAgeSeconds": 3600
  }
]

04Keep secrets out of the page

Public files need no key. Private files are handed out as short-lived presigned URLs by a server you control. A key typed into config.js ships to every visitor.

05Measure what compression saves

The second example compresses eight files in your browser with the built-in CompressionStream and prints both sizes. Text files shrink to between 12 % and 41 %.

Try it yourself

  1. 1

    Change the base

    In the first example add ["a bucket, no end slash", "https://YOUR-BUCKET.example/lumen"] to bases. The URL printed has exactly one slash before stations.csv, and the result is TypeError: Failed to fetch (no such bucket).

  2. 2

    Compress your own text

    In the second example add "daily-dirty.csv" to files. It is 1,215 bytes and gzip makes it 524 (43 %): small files compress worse than big ones.

Code & diagrams

Where the data lives: one value, three bases New tab

The placeholder bucket does not exist, so the request fails like any blocked address. The naive join without a slash is a 404: a typo, not a network problem.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

local files (the course default)          25 stations loaded
local, base typed without the end slash   25 stations loaded
a public bucket (PLACEHOLDER name)        TypeError: Failed to fetch
Naive join: /data/lumenstations.csv
  -> 404 Not Found
How much does gzip save? New tab

Classic script (no import), so the code sits inside an async function. Sizes are the real bytes of the course files.

Preview

Press Run to see the page.

Console

Nothing logged yet.

Expected console output

stations.csv            1414 bytes     555 gzipped   39%
daily.csv              13406 bytes    4587 gzipped   34%
hourly-june.csv        19334 bytes    3444 gzipped   18%
trips-10k.csv         487313 bytes  146324 gzipped   30%
bikes.json             24053 bytes    2790 gzipped   12%
routes.json             3932 bytes     661 gzipped   17%
ardenvale.topojson      2684 bytes    1089 gzipped   41%
world-110m.json       107761 bytes   38423 gzipped   36%

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Load from a bucket with no CORS rule

Set DATA_URL to a bucket address that has no CORS configuration and run the dashboard at http://localhost:5173. (Shown, not live: this preview cannot reach other sites.)

terminal
$ Console (Chromium, measured against a local test server)
── what you'll see ──
Access to fetch at 'http://localhost:8091/stations.csv' from origin 'http://localhost:5173' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
Failed to load resource: net::ERR_FAILED
TypeError: Failed to fetch

Myth vs fact

Myth

CORS is a bug in my code.

Fact

It is the server's decision, enforced by the browser. The fix is a header on the storage, not a code change in the page.

Myth

I can hide my access key in the JavaScript bundle.

Fact

Everything you ship is readable. Use presigned URLs from a server you control.

Myth

Making a bucket public is the same as setting up CORS.

Fact

Public read lets anyone fetch the file. CORS decides which web pages may read it from JavaScript. You usually need both for public data.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

  • ▸

    A CORS rule can hide behind a CDN cache: the first response caches without the right header, and later requests from the allowed origin fail. If the server varies the header by origin, it must send Vary: Origin.

  • ▸

    Presigned URLs carry the signer's permissions and expire. Keep the lifetime short, scope it to one object, and log who asked. A leaked one works until it expires.

Remember this

  1. 1

    The problem this solves: the data will outgrow the project folder, and many people, not only you, will open the dashboard. Cloud object storage such as Amazon S3 or Google Cloud Storage (GCS) keeps files in a bucket (a named container) and serves each at a URL. The dashboard stays a static site and the data updates without a new build.

  2. 2

    One config value. Every URL is built from DATA_URL in src/utils/config.js. The first example builds URLs with a dataUrl(base, file) helper that puts exactly one slash between base and file, then loads stations.csv from three bases: the local folder (25 stations loaded), the same without its end slash (still 25) and a placeholder bucket address (TypeError: Failed to fetch, because that bucket does not exist). The naive base + file gives /data/lumenstations.csv, a 404. Names like YOUR-BUCKET here are placeholders: this course does not use a real bucket.

  3. 3

    CORS. A page on https://dashboard.example reading data from https://bucket.example is a cross-origin request. The browser sends the request but hands the answer to your code only if the storage answers with an Access-Control-Allow-Origin header that allows your page's origin. If not, the Console says (measured in Chromium against a local test server with no such header) Access to fetch at 'http://localhost:8091/stations.csv' from origin 'http://localhost:5173' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. Your code only sees TypeError: Failed to fetch, the same as an offline network. With a header that names a different site, the message ends …has a value 'https://other.example' that is not equal to the supplied origin.

  4. 4

    Fixing CORS is a bucket setting, not frontend code. The bucket owner adds a CORS rule that allows GET from the dashboard's origin. Allow only the origins you serve, not *, unless the data is public anyway. In S3 the rule is a small JSON document (shown in the code block below). During development the rule must include http://localhost:5173.

  5. 5

    The same in Google Cloud Storage. A GCS file is served at https://storage.googleapis.com/<bucket>/<key>, for example https://storage.googleapis.com/YOUR-BUCKET/lumen/daily.csv. Public read is a permission: grant the role Storage Object Viewer to the principal allUsers (read only: no writes, no other buckets). Note that this role also lets anyone list the file names in this one bucket; if the names should stay private, grant Storage Legacy Object Reader instead, which allows reading a file whose name you know but not listing. CORS is a JSON file with the keys origin, method, responseHeader and maxAgeSeconds, applied with gcloud storage buckets update gs://YOUR-BUCKET --cors-file=cors.json (second snippet in the walkthrough). Hectal itself serves this course's Lumen files from a GCS bucket this way. One trap we met while setting it up: GCS caches public files for an hour by default (Cache-Control: public, max-age=3600). A file fetched before the CORS rule was added kept coming back without Access-Control-Allow-Origin until that cached copy expired, while files fetched for the first time had the header. After a CORS change, test with a file nobody has fetched yet, or wait out the cache.

  6. 6

    Public read-only data. For data anyone may see (like a city's open data), the bucket (or better, a CDN in front of it) allows anonymous GET and nothing else: no listing, no write, no delete. Make this choice on purpose and never put personal data in a public bucket. Review what is in it before switching public access on.

  7. 7

    Cache-Control and size. The Cache-Control header tells the browser how long it may reuse a file: max-age=3600 means one hour without asking again. A static yearly export can cache for long; a live snapshot (status.json) should not. Big files are sent compressed: gzip shrinks text a lot. The second example compresses the course files in the browser: trips-10k.csv 487,313 → 146,324 bytes (30 %), daily.csv 13,406 → 4,587 (34 %), bikes.json 24,053 → 2,790 (12 %). Serve compressed (CDNs usually do it for you) and keep very big tables out of the browser anyway (Phase 12).

  8. 8

    Private data and presigned URLs. For data not meant for everyone, the dashboard must never hold the cloud account's access key: anything in frontend code is readable by every visitor. Instead a small server you control, which holds the secret, signs a presigned URL (a normal URL with a signature and an expiry time, for example 5 minutes, for exactly one file) and hands it to the logged-in user. The browser then fetches the file directly. If a key was ever pasted into frontend code, treat it as leaked: revoke it and create a new one.

Explain it without notes

01

Why does moving the data to a bucket change only one line of code?

02

What does the browser do when a response has no Access-Control-Allow-Origin header, and what does your code see?

03

Why must a secret key never appear in frontend code, and what do you use instead?

Practice

01

Write dataUrl(base, file) so that dataUrl("/data/lumen", "/daily.csv") and dataUrl("/data/lumen/", "daily.csv") both give /data/lumen/daily.csv.

02

A teammate sees Failed to fetch for the bucket URL but the file opens in a new tab. Name the likely cause and the first place to look.

Trade-offs

  • ↔

    A bucket plus CDN scales and needs no server, at the cost of CORS and cache settings to get right. Local public/ files are simplest for a demo and for the course.

Done when you can

  • I can change where the data lives by editing one config value.

  • I can recognise a CORS error in the Console and say what fixes it.

  • I know why access keys never go into frontend code.

  • I can say what Cache-Control and gzip do for a big file.