Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
37cd08f
run successfully
GuoTeng-ECNU May 21, 2024
68f7044
need debug
GuoTeng-ECNU Jun 11, 2024
d00e49e
update HeatBuckets
GuoTeng-ECNU Jun 12, 2024
99b0259
update
GuoTeng-ECNU Jun 12, 2024
0b4456f
commit
GuoTeng-ECNU Jul 1, 2024
d01916e
update
GuoTeng-ECNU Jul 1, 2024
d191a5e
update hotness estimate
GuoTeng-ECNU Jul 13, 2024
2ae380c
Merge pull request #1 from dase955/gt-dev
GuoTeng-ECNU Jul 13, 2024
1f1110d
add classifier demo
GuoTeng-ECNU Jul 23, 2024
f4611f0
add basic filter unit support
DayuanX Jul 24, 2024
875787f
temp
GuoTeng-ECNU Jul 24, 2024
cc6dfeb
Merge pull request #2 from dase955/wqy-dev
GuoTeng-ECNU Jul 24, 2024
c3f3179
still add clf_model
GuoTeng-ECNU Jul 25, 2024
bfde765
Merge pull request #3 from dase955/gt-dev
GuoTeng-ECNU Jul 25, 2024
4cfca28
add flag for previous commit
DayuanX Jul 25, 2024
4429032
Merge branch 'wqy-dev' of github.com:dase955/WaLSM into wqy-dev
DayuanX Jul 25, 2024
e97e4fa
fix compile error
GuoTeng-ECNU Jul 25, 2024
74fba2e
Merge pull request #4 from dase955/wqy-dev
GuoTeng-ECNU Jul 25, 2024
b0eeb9c
modify compile command
GuoTeng-ECNU Jul 25, 2024
8611896
Merge pull request #5 from dase955/gt-dev
GuoTeng-ECNU Jul 25, 2024
441b384
YCSB compile modify
GuoTeng-ECNU Jul 25, 2024
99c517b
fix
GuoTeng-ECNU Jul 25, 2024
3dd920c
Merge pull request #6 from dase955/gt-dev
GuoTeng-ECNU Jul 25, 2024
d69338d
bugfix & warning fix
DayuanX Jul 25, 2024
7ed393c
Merge branch 'wqy-dev' of github.com:dase955/WaLSM into wqy-dev
DayuanX Jul 25, 2024
056808f
debug models
GuoTeng-ECNU Jul 26, 2024
6de4dfc
Merge pull request #7 from dase955/wqy-dev
GuoTeng-ECNU Jul 26, 2024
c37fd6f
Merge pull request #8 from dase955/gt-dev
GuoTeng-ECNU Jul 26, 2024
4024c1b
have bug when using python lightgbm
GuoTeng-ECNU Jul 26, 2024
f33b837
try seperate lgb from WaLSM side
GuoTeng-ECNU Jul 30, 2024
67692e7
add socket example
GuoTeng-ECNU Jul 31, 2024
6f300be
update clfmodel
GuoTeng-ECNU Aug 1, 2024
9edd579
remove python.h deps
GuoTeng-ECNU Aug 1, 2024
ee8256f
debug clfmodel completed
GuoTeng-ECNU Aug 3, 2024
fb7bde0
ready to debug greedy algo
GuoTeng-ECNU Aug 5, 2024
6114c25
filter greedy algo done
GuoTeng-ECNU Aug 6, 2024
c49efe5
write filter cache heap done
GuoTeng-ECNU Aug 7, 2024
1b9ff9b
temp save
GuoTeng-ECNU Aug 8, 2024
1fa11d3
ready to debug
GuoTeng-ECNU Aug 9, 2024
82e9ede
udpate bits-per-key
GuoTeng-ECNU Aug 10, 2024
99760d3
filter cache heap manager debug start
GuoTeng-ECNU Aug 10, 2024
f3748ba
filter cache heap done
GuoTeng-ECNU Aug 12, 2024
87de64f
Merge pull request #9 from dase955/gt-dev
GuoTeng-ECNU Aug 12, 2024
2523f9e
filter cache item ready to write
GuoTeng-ECNU Aug 13, 2024
c69a666
Merge pull request #10 from dase955/gt-dev
GuoTeng-ECNU Aug 13, 2024
a74f50c
add FilterCache
GuoTeng-ECNU Aug 14, 2024
3f6ed4a
temp work
GuoTeng-ECNU Aug 16, 2024
169cf17
temp work
GuoTeng-ECNU Aug 16, 2024
238d13f
temp work
GuoTeng-ECNU Aug 18, 2024
e396e48
temp work
GuoTeng-ECNU Aug 18, 2024
f61a19b
temp work
GuoTeng-ECNU Aug 23, 2024
781ecaf
temp
GuoTeng-ECNU Aug 24, 2024
e51251d
need background thread pool to do FilterCacheManager
GuoTeng-ECNU Aug 27, 2024
8ec261e
Merge pull request #11 from dase955/gt-dev
GuoTeng-ECNU Aug 27, 2024
b6fdd08
update
GuoTeng-ECNU Aug 28, 2024
706fee3
Merge pull request #12 from dase955/gt-dev
GuoTeng-ECNU Aug 28, 2024
f2f6d4f
thread pool demo
GuoTeng-ECNU Aug 30, 2024
c6afb39
try FilterCacheClient
GuoTeng-ECNU Aug 30, 2024
5a5a7cb
update
GuoTeng-ECNU Sep 2, 2024
35da00f
Merge pull request #13 from dase955/gt-dev
GuoTeng-ECNU Sep 2, 2024
e5a93c7
use universal compactions for tests
DayuanX Sep 8, 2024
a40fcfd
Merge pull request #14 from dase955/wqy-dev
GuoTeng-ECNU Sep 9, 2024
bbc84c4
add level 0 base count
GuoTeng-ECNU Sep 11, 2024
8bb9d66
Merge branch 'gt-dev' of github.com:dase955/WaLSM into gt-dev
GuoTeng-ECNU Sep 11, 2024
7579786
some modification
GuoTeng-ECNU Sep 12, 2024
11a884c
expose filter cache object to FilterReader
GuoTeng-ECNU Sep 12, 2024
458122f
add some information structures
GuoTeng-ECNU Sep 14, 2024
94b7c3c
small fix
GuoTeng-ECNU Sep 15, 2024
8ff5c50
small fix
GuoTeng-ECNU Sep 17, 2024
8a3e491
read flush
GuoTeng-ECNU Sep 22, 2024
8f83379
flush note
GuoTeng-ECNU Sep 23, 2024
9db37f8
update Flush note
GuoTeng-ECNU Sep 23, 2024
9e1d1ae
update note
GuoTeng-ECNU Sep 24, 2024
e127701
update note
GuoTeng-ECNU Sep 25, 2024
e4817cc
consider delete compaction and terival compaction
GuoTeng-ECNU Sep 25, 2024
ebfcd22
add note
GuoTeng-ECNU Sep 26, 2024
0244ec3
still need consider compaction info recorders' process?
GuoTeng-ECNU Sep 27, 2024
a1e2e6a
still need complete normal compaction
GuoTeng-ECNU Sep 29, 2024
7f4096f
need debug
GuoTeng-ECNU Sep 29, 2024
a6a0da5
need debug
GuoTeng-ECNU Sep 29, 2024
e29215f
fix some bugs
GuoTeng-ECNU Dec 24, 2024
0614317
YCSB update
GuoTeng-ECNU Jan 14, 2025
8b59246
YCSB set Parallelism
GuoTeng-ECNU Jan 14, 2025
be8ad8a
modify
GuoTeng-ECNU Mar 11, 2025
46a7794
prevent multi-updating heat buckets in short time
GuoTeng-ECNU Mar 25, 2025
b02571f
sync adjustment heap after a training period
GuoTeng-ECNU Mar 26, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -91,4 +91,16 @@ clang-format-diff.py
.py3/

cmake-build-debug
cmake-build-release
cmake-build-release

*_example
inode_vptrs
.cache/
.conda/
*.txt
__pycache__/
*.csv

include/csv2/
debug.*
*.log
8 changes: 8 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -256,6 +256,7 @@ endif(CMAKE_SYSTEM_PROCESSOR MATCHES "aarch64|AARCH64")
set(CMAKE_CXX_FLAGS "-mavx2 -lpmem ${CMAKE_CXX_FLAGS}")
add_definitions(-DART)
add_definitions(-DUSE_PMEM)
add_definitions(-DART_PLUS)

option(PORTABLE "build a portable binary" OFF)
option(FORCE_SSE42 "force building with SSE4.2, even when PORTABLE=ON" OFF)
Expand Down Expand Up @@ -589,6 +590,13 @@ set(SOURCES
db/art/art_node.cc
db/art/compactor.cc
db/art/global_memtable.cc
db/art/heat_buckets.cc
db/art/clf_model.cc
db/art/filter_cache_heap.cc
db/art/filter_cache_item.cc
db/art/filter_cache.cc
db/art/filter_cache_client.cc
db/art/greedy_algo.cc
db/art/heat_group.cc
db/art/lock.cc
db/art/logger.cc
Expand Down
7 changes: 7 additions & 0 deletions TARGETS
Original file line number Diff line number Diff line change
Expand Up @@ -447,6 +447,13 @@ cpp_library(
"db/art/art_node.cc"
"db/art/compactor.cc"
"db/art/global_memtable.cc",
"db/art/heat_buckets.cc",
"db/art/clf_model.cc",
"db/art/filter_cache_heap.cc",
"db/art/filter_cache_item.cc",
"db/art/filter_cache.cc",
"db/art/filter_cache_client.cc",
"db/art/greedy_algo.cc",
"db/art/heat_group.cc",
"db/art/lock.cc",
"db/art/logger.cc",
Expand Down
10 changes: 8 additions & 2 deletions YCSB/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,19 @@
#---------------------build config-------------------------

DEBUG_BUILD ?= 0
EXTRA_CXXFLAGS ?= -I../include -I../include/rocksdb
EXTRA_LDFLAGS ?= -L../ -lpmem -ldl -lz -lbz2 -lsnappy -llz4 -lzstd
EXTRA_CXXFLAGS ?= -I../include -I../include/rocksdb
EXTRA_LDFLAGS ?= -L../ -lpmem -ldl

BIND_ROCKSDB ?= 1
BIND_LEVELDB ?= 0
BIND_LMDB ?= 0

EXTRA_LDFLAGS += -lstdc++
EXTRA_LDFLAGS += -lsocket++
# EXTRA_LDFLAGS += -lpython3.12
# EXTRA_CXXFLAGS += -I$(PYTHON_INCLUDE_PATH)
# EXTRA_CXXFLAGS += -L$(PYTHON_LIBRARY_PATH)

#----------------------------------------------------------

ifeq ($(DEBUG_BUILD), 1)
Expand Down
10 changes: 0 additions & 10 deletions YCSB/leveldb/leveldb_db.cc
Original file line number Diff line number Diff line change
Expand Up @@ -151,16 +151,6 @@ void LeveldbDB::GetOptions(const utils::Properties &props, leveldb::Options *opt
}
}

void LeveldbDB::SerializeRow(const std::vector<Field> &values, std::string *data) {
for (const Field &field : values) {
uint32_t len = field.name.size();
data->append(reinterpret_cast<char *>(&len), sizeof(uint32_t));
data->append(field.name.data(), field.name.size());
len = field.value.size();
data->append(reinterpret_cast<char *>(&len), sizeof(uint32_t));
data->append(field.value.data(), field.value.size());
}
}

void LeveldbDB::DeserializeRowFilter(std::vector<Field> *values, const std::string &data,
const std::vector<std::string> &fields) {
Expand Down
8 changes: 5 additions & 3 deletions YCSB/rocksdb/rocksdb.properties
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
rocksdb.dbname=/tmp/tmp_data/db_nvm_l0
rocksdb.nvm_path=/mnt/chen/nodememory
rocksdb.dbname=/mnt/walsm/tmp/tmp_data/db_test_art
rocksdb.nvm_path=/mnt/walsm/node_memory
rocksdb.format=single
rocksdb.destroy=false

Expand All @@ -23,5 +23,7 @@ rocksdb.cache_size=8388608
rocksdb.compressed_cache_size=0
rocksdb.bloom_bits=0

# set total_threads to 32, see rocksdb_db.cc
rocksdb.increase_parallelism=true
rocksdb.optimize_level_style_compaction=true
# rocksdb.optimize_level_style_compaction=true
rocksdb.optimize_universal_style_compaction=true
19 changes: 18 additions & 1 deletion YCSB/rocksdb/rocksdb_db.cc
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
#include <rocksdb/status.h>
#include <rocksdb/utilities/options_util.h>
#include <rocksdb/write_batch.h>
#include <fstream>

namespace {
const std::string PROP_NAME = "rocksdb.dbname";
Expand Down Expand Up @@ -98,6 +99,9 @@ namespace {
const std::string PROP_OPTIMIZE_LEVELCOMP = "rocksdb.optimize_level_style_compaction";
const std::string PROP_OPTIMIZE_LEVELCOMP_DEFAULT = "false";

const std::string PROP_OPTIMIZE_UNIVERSALCOMP = "rocksdb.optimize_universal_style_compaction";
const std::string PROP_OPTIMIZE_UNIVERSALCOMP_DEFAULT = "false";

const std::string PROP_OPTIONS_FILE = "rocksdb.optionsfile";
const std::string PROP_OPTIONS_FILE_DEFAULT = "";

Expand Down Expand Up @@ -351,11 +355,14 @@ void RocksdbDB::GetOptions(const utils::Properties &props, rocksdb::Options *opt
opt->table_factory.reset(rocksdb::NewBlockBasedTableFactory(table_options));

if (props.GetProperty(PROP_INCREASE_PARALLELISM, PROP_INCREASE_PARALLELISM_DEFAULT) == "true") {
opt->IncreaseParallelism();
opt->IncreaseParallelism(32);
}
if (props.GetProperty(PROP_OPTIMIZE_LEVELCOMP, PROP_OPTIMIZE_LEVELCOMP_DEFAULT) == "true") {
opt->OptimizeLevelStyleCompaction();
}
if (props.GetProperty(PROP_OPTIMIZE_UNIVERSALCOMP, PROP_OPTIMIZE_UNIVERSALCOMP_DEFAULT) == "true") {
opt->OptimizeUniversalStyleCompaction();
}
}
}

Expand Down Expand Up @@ -424,6 +431,12 @@ DB::Status RocksdbDB::ReadSingle(const std::string &table, const std::string &ke
std::vector<Field> &result) {
std::string data;
rocksdb::Status s = db_->Get(rocksdb::ReadOptions(), key, &data);
#ifdef GEN_WORKLOAD
std::fstream f;
f.open("../workload/workload", std::ios::out | std::ios::app);
f << key <<std::endl;
f.close();
#endif
if (s.IsNotFound()) {
return kNotFound;
} else if (!s.ok()) {
Expand Down Expand Up @@ -461,6 +474,7 @@ DB::Status RocksdbDB::ScanSingle(const std::string &table, const std::string &ke

DB::Status RocksdbDB::UpdateSingle(const std::string &table, const std::string &key,
std::vector<Field> &values) {
/*
std::string data;
rocksdb::Status s = db_->Get(rocksdb::ReadOptions(), key, &data);
if (s.IsNotFound()) {
Expand Down Expand Up @@ -491,6 +505,9 @@ DB::Status RocksdbDB::UpdateSingle(const std::string &table, const std::string &
throw utils::Exception(std::string("RocksDB Put: ") + s.ToString());
}
return kOK;
*/
// use insert, not read-modify-write
return InsertSingle(table, key, values);
}

DB::Status RocksdbDB::MergeSingle(const std::string &table, const std::string &key,
Expand Down
16 changes: 16 additions & 0 deletions YCSB/workloads/workloadt
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Yahoo! Cloud System Benchmark
# Workload T: For Debug


recordcount=5000000
operationcount=2200000
workload=com.yahoo.ycsb.workloads.CoreWorkload

readallfields=true

readproportion=1
updateproportion=0
scanproportion=0
insertproportion=0

requestdistribution=zipfian
13 changes: 12 additions & 1 deletion build_tools/build_detect_platform
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,13 @@ if test -z "$OUTPUT"; then
exit 1
fi

ROCKSDB_DISABLE_SNAPPY=1
ROCKSDB_DISABLE_ZLIB=1
ROCKSDB_DISABLE_BZIP=1
ROCKSDB_DISABLE_LZ4=1
ROCKSDB_DISABLE_ZSTD=1
ROCKSDB_DISABLE_GFLAGS=1

# we depend on C++11
PLATFORM_CXXFLAGS="-std=c++11"
# we currently depend on POSIX platform
Expand Down Expand Up @@ -272,7 +279,11 @@ JAVA_LDFLAGS="$PLATFORM_LDFLAGS"
JAVA_STATIC_LDFLAGS="$PLATFORM_LDFLAGS"
JAVAC_ARGS="-source 7"

COMMON_FLAGS="$COMMON_FLAGS -DUSE_PMEM -DART"
COMMON_FLAGS="$COMMON_FLAGS -DUSE_PMEM -DART -DART_PLUS"
COMMON_FLAGS="$COMMON_FLAGS -lstdc++ -lsocket++"
# COMMON_FLAGS="$COMMON_FLAGS -lstdc++ -lpython3.12"
# COMMON_FLAGS="$COMMON_FLAGS -I$PYTHON_INCLUDE_PATH"
# COMMON_FLAGS="$COMMON_FLAGS -L$PYTHON_LIBRARY_PATH"
PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lpmem"
JAVA_LDFLAGS="$JAVA_LDFLAGS -lpmem"

Expand Down
Loading