1Start all services

From the project root, start the containers in the background:

docker compose up -d

Check their status:

docker ps

2Work with RustFS object storage

RustFS provides S3-compatible object storage for this lab.

1. Open the RustFS console: http://localhost:9001.
2. Sign in using the credentials configured for your environment.
3. Create these buckets (if they do not already exist):

4. Open bucket 01-raw and create or navigate to the prefix people/.
5. Upload inputs/people1.csv so its object path is 01-raw/people/people1.csv.

docker compose --env-file .env -f aws-cli/compose.yaml build
docker compose --env-file .env -f aws-cli/compose.yaml run --rm aws-cli

aws --endpoint-url http://rustfs:9000 s3 mb s3://01-raw
aws --endpoint-url http://rustfs:9000 s3 mb s3://02-curated
aws --endpoint-url http://rustfs:9000 s3 mb s3://03-processed

aws --endpoint-url http://rustfs:9000 s3 ls

aws --endpoint-url http://rustfs:9000 s3 cp ./inputs/people1.csv s3://01-raw/people/people1.csv
aws --endpoint-url http://rustfs:9000 s3 ls s3://01-raw/people/

exit

3Query files without a metadata catalog

In this part, Spark reads the CSV file directly from object storage.

Start the interactive Spark SQL shell:

docker compose exec --user spark spark /opt/spark/bin/spark-sql

Enable column headers in query output:

SET spark.sql.cli.print.header=true;

Query the CSV file directly. The example uses people1.csv, uploaded in the previous step:

SELECT *
FROM csv.`s3a://01-raw/people/people1.csv`;

4Work with a metadata catalog (Hive Metastore)

Here, a SQL script registers a table over the raw CSV data. The table definition is stored in Hive Metastore; the source CSV remains in object storage.

Register the table and perform initial data load:

SOURCE /sql/01_create_people.sql;

Verify:

SELECT * FROM lake_demo.people;

Add another source file

1. In the RustFS console, upload inputs/people2.csv to bucket 01-raw, under prefix people/: 01-raw/people/people2.csv.
2. Verify (REFRESH TABLE refreshes cached metadata):

REFRESH TABLE lake_demo.people;
SELECT * FROM lake_demo.people;

Explore the catalog

SHOW TABLES IN lake_demo;
DESCRIBE EXTENDED lake_demo.people;

When finished, exit Spark SQL:

quit;

5Work with Apache Iceberg

In this part, Spark creates and queries an Iceberg table using the configured Iceberg catalog.

Start the Iceberg-enabled Spark SQL shell:

docker compose exec --user spark spark-iceberg /opt/spark/bin/spark-sql

Enable column headers:

SET spark.sql.cli.print.header=true;

Create an Iceberg table

Run the script that creates the Iceberg table from the Hive Metastore catalog table:

SOURCE /sql/02_create_people_iceberg.sql;

Verify the data and inspect the catalog entry:

SELECT * FROM iceberg.lakehouse_demo.people_iceberg;

SHOW TABLES IN iceberg.lakehouse_demo;

DESCRIBE TABLE EXTENDED iceberg.lakehouse_demo.people_iceberg;

Inspect snapshots and insert a row

List the table's snapshots:

SOURCE /sql/03_get_snapshots.sql;

Insert a row using the supplied script:

SOURCE /sql/04_insert_people.sql;

Verify the updated table:

SELECT * FROM iceberg.lakehouse_demo.people_iceberg;

List snapshots again and compare the results:

SOURCE /sql/03_get_snapshots.sql;

Finally, query an earlier version of the table:

SOURCE /sql/05_get_old_data.sql;

6Useful commands

Stop and remove containers

docker compose down

This stops and removes the Compose containers and network. Named volumes and persisted object-storage data are generally retained unless you explicitly remove them.

Restart one service

docker compose restart spark

Recreate one service after changing its Compose configuration

docker compose up -d --force-recreate spark

Remove the Hive database volume — destructive

docker volume rm data_lakehouse_demo_hive_data_v3

If Docker reports that the volume is in use, stop the relevant service first. Confirm the actual volume name with docker volume ls.