write_and_read_streams.cpp File Reference

#include <algorithm>
#include <cstdlib>
#include <filesystem>
#include <fstream>
#include <iostream>
#include <random>
#include <vector>
#include <sparrow_ipc/deserialize.hpp>
#include <sparrow_ipc/memory_output_stream.hpp>
#include <sparrow_ipc/serializer.hpp>
#include <sparrow/record_batch.hpp>

Include dependency graph for write_and_read_streams.cpp:

Go to the source code of this file.

Namespaces
namespace	utils

Functions
std::mt19937	gen (rd())
sp::record_batch	utils::create_random_record_batch (size_t num_rows)
	Helper function to create a record batch with the same schema but random values All batches have: int32 column, float column, bool column, and string column.
bool	utils::verify_batches_match (const std::vector< sp::record_batch > &original_batches, const std::vector< sp::record_batch > &deserialized_batches)
	Verify that two sets of record batches are identical Returns true if all batches match, false otherwise.
std::vector< sp::record_batch >	create_record_batches (size_t num_batches, size_t rows_per_batch)
	Create multiple record batches with the same schema but random values.
std::vector< uint8_t >	serialize_batches_to_stream (const std::vector< sp::record_batch > &batches)
	Serialize record batches to a stream.
std::vector< sp::record_batch >	deserialize_stream_to_batches (const std::vector< uint8_t > &stream_data)
	Deserialize stream back to record batches.
void	demonstrate_serialization_methods (const std::vector< sp::record_batch > &batches, const std::vector< uint8_t > &batch_stream_data)
	Demonstrate individual vs batch serialization.
bool	verify_schema_consistency (const std::vector< sp::record_batch > &batches)
	Verify schema consistency across all batches.
void	read_and_display_test_file ()
	Read and display a primitive stream file from test resources.
int	main ()

Variables
const std::filesystem::path	arrow_testing_data_dir = ARROW_TESTING_DATA_DIR
const std::filesystem::path	tests_resources_files_path
std::random_device	rd

Function Documentation

create_record_batches()

std::vector< sp::record_batch > create_record_batches	(	size_t	num_batches,
		size_t	rows_per_batch )

Create multiple record batches with the same schema but random values.

Definition at line 163 of file write_and_read_streams.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

demonstrate_serialization_methods()

void demonstrate_serialization_methods	(	const std::vector< sp::record_batch > &	batches,
		const std::vector< uint8_t > &	batch_stream_data )

Demonstrate individual vs batch serialization.

Definition at line 226 of file write_and_read_streams.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

deserialize_stream_to_batches()

std::vector< sp::record_batch > deserialize_stream_to_batches

(

const std::vector< uint8_t > &

stream_data

)

Deserialize stream back to record batches.

Definition at line 210 of file write_and_read_streams.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

gen()

std::mt19937 gen

(

rd()

)

Here is the caller graph for this function:

main()

int main

(

)

Definition at line 365 of file write_and_read_streams.cpp.

Here is the call graph for this function:

read_and_display_test_file()

void read_and_display_test_file

(

)

Read and display a primitive stream file from test resources.

Definition at line 318 of file write_and_read_streams.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

serialize_batches_to_stream()

std::vector< uint8_t > serialize_batches_to_stream

(

const std::vector< sp::record_batch > &

batches

)

Serialize record batches to a stream.

Definition at line 189 of file write_and_read_streams.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

verify_schema_consistency()

bool verify_schema_consistency

(

const std::vector< sp::record_batch > &

batches

)

Verify schema consistency across all batches.

Definition at line 264 of file write_and_read_streams.cpp.

Here is the caller graph for this function:

Variable Documentation

arrow_testing_data_dir

const std::filesystem::path arrow_testing_data_dir = ARROW_TESTING_DATA_DIR

Definition at line 15 of file write_and_read_streams.cpp.

rd

std::random_device rd

Definition at line 23 of file write_and_read_streams.cpp.

tests_resources_files_path

const std::filesystem::path tests_resources_files_path

Initial value:

= arrow_testing_data_dir / "data" / "arrow-ipc-stream"
                                                         / "integration" / "cpp-21.0.0"

Definition at line 16 of file write_and_read_streams.cpp.