| 标题 |
指标名称 |
指标单位 |
指标含义 |
| 当前App个数 |
app_num_with_node |
Count |
当前在ShuffleServer上存储数据的 application 的数量 |
| 缓冲区 |
allocated_buffer_size |
Bytes |
已分配出来存储数据的内存量 |
| |
in_flush_buffer_size |
Bytes |
待 flush 到磁盘的内存量 |
| |
used_buffer_size |
Bytes |
内存用来存储 shuffle 数据的用量,上限由 rss.server.buffer.capacity 确定 |
| |
read_used_buffer_size |
Bytes |
用来缓存 shuffle read 数据的用量,上限由 rss.server.read.buffer.capacity 确定 |
| |
total_expired_preAllocated_buffer_num |
Count |
过期的preAllocatedBuffer个数 |
| |
require_buffer_count |
Count |
申请preAllocatedBuffer个数 |
| 数据 |
total_received_data |
Bytes |
ShuffleServer接收到的数据量 |
| |
total_write_data |
Bytes |
ShuffleServer Flush 到磁盘/远端存储的数据量 |
| |
total_delete_data |
Bytes |
删除Application 磁盘/HDFS存储大小 |
| 数据删除数量 |
total_delete_file_num |
Count |
删除文件总数 |
| Direct内存 |
used_direct_memory_size |
Bytes |
direct memory使用大小 |
| |
used_by_netty |
Bytes |
Netty使用direct memory大小 |
| |
used_by_grpc_netty |
Bytes |
grpc_netty使用direct memory大小 |
| 事件 |
event_queue_size |
Count |
当前ShuffleServer flush 队列中的event 数量 |
| |
total_failed_written_event_num |
Count |
event 持久化失败的数量 |
| |
total_dropped_event_num |
Count |
由于 application 不存在或者flush 超时导致event被丢掉的数量 |
| Flush数量 |
total_flush_file_num |
Count |
ShuffleServer Flush 到磁盘/远端存储的文件总数 |
| Flush数量 |
flush_file_num |
Count |
当前正在Flush的文件个数 |
| |
total_write_num |
Count |
ShuffleServer Flush 到磁盘/远端存储的次数 |
| |
storage_total_write_local |
Count |
写本地存储的总次数 |
| |
storage_retry_write_local |
Count |
重试写本地存储的次数 |
| |
storage_failed_write_local |
Count |
写本地存储失败的次数 |
| |
storage_success_write_local |
Count |
写本地存储成功的次数 |
| |
local_file_event_flush_num |
Count |
Flush到本地存储次数 |
| |
hadoop_event_flush_num |
Count |
Flush到Hadoop次数 |
| |
flush_exceed_high_water_count |
Count |
buffer中数据量超过水位线导致Flush的次数(水位线由rss.server.memory.shuffle.highWaterMark.percentage控制) |
| |
flush_exceed_buffer_threshold_count |
Count |
buffer中数据量大小超过阈值导致Flush的次数(阈值由rss.server.single.buffer.flush.threshold控制) |
| |
flush_exceed_block_num_count |
Count |
buffer中block数超过阈值导致Flush的次数(阈值由rss.server.single.buffer.flush.blocksNumberThreshold控制) |
| Flush数据量 |
total_hdfs_write_data |
Bytes |
写到HDFS上的数据量 |
| Flush线程池 |
localfile_flush_thread_pool_queue_size |
Count |
Flush到本地的线程池队列中排队任务数 |
| |
read_local_data_file_thread_num |
Count |
当前读本地存储数据线程数 |
| |
read_local_index_file_thread_num |
Count |
当前读本地存储索引线程数 |
| |
read_memory_data_thread_num |
Count |
当前读内存数据线程数 |
| |
localfile_writing_thread_num |
Count |
Flush到本地线程数量 |
| |
hdfs_writing_thread_num |
Count |
Flush到HDFS线程数量 |
| gRPC Process时延 |
send_shuffle_data_latency_50 |
s |
grpc send_shuffle_data process 50分位时延 |
| |
send_shuffle_data_latency_90 |
s |
grpc send_shuffle_data process 90分位时延 |
| |
send_shuffle_data_latency_99 |
s |
grpc send_shuffle_data process 99分位时延 |
| |
send_shuffle_data_latency_sum |
s |
grpc send_shuffle_data process 总时延 |
| |
get_local_shuffle_data_latency_50 |
s |
grpc get_local_shuffle_data process 50分位时延 |
| |
get_local_shuffle_data_latency_90 |
s |
grpc get_local_shuffle_data process 90分位时延 |
| |
get_local_shuffle_data_latency_99 |
s |
grpc get_local_shuffle_data process 99分位时延 |
| |
get_local_shuffle_data_latency_sum |
s |
grpc get_local_shuffle_data process 总时延 |
| |
get_memory_shuffle_data_latency_50 |
s |
gRPC get_memory_shuffle_data process 50分位时延 |
| |
get_memory_shuffle_data_latency_90 |
s |
gRPC get_memory_shuffle_data process 90分位时延 |
| |
get_memory_shuffle_data_latency_99 |
s |
gRPC get_memory_shuffle_data process 99分位时延 |
| |
get_memory_shuffle_data_latency_sum |
s |
gRPC get_memory_shuffle_data process 总时延 |
| |
get_multipart_shuffle_res_latency_50 |
s |
gRPC get_shuffle_result_for_multi_part process 50分位时延 |
| |
get_multipart_shuffle_res_latency_90 |
s |
gRPC get_shuffle_result_for_multi_part process 90分位时延 |
| |
get_multipart_shuffle_res_latency_99 |
s |
gRPC get_shuffle_result_for_multi_part process 99分位时延 |
| |
get_multipart_shuffle_res_latency_sum |
s |
gRPC get_shuffle_result_for_multi_part process总时延 |
| |
report_shuffle_result_latency_50 |
s |
gRPC report_shuffle_result process 50分位时延 |
| |
report_shuffle_result_latency_90 |
s |
gRPC report_shuffle_result process 90分位时延 |
| |
report_shuffle_result_latency_99 |
s |
gRPC report_shuffle_result process 99分位时延 |
| |
report_shuffle_result_latency_sum |
s |
gRPC report_shuffle_result process 总时延 |
| |
get_sorted_shuffle_data_latency_50 |
s |
gRPC get_sorted_shuffle_data process 50分位时延 |
| |
get_sorted_shuffle_data_latency_90 |
s |
gRPC get_sorted_shuffle_data process 90分位时延 |
| |
get_sorted_shuffle_data_latency_99 |
s |
gRPC get_sorted_shuffle_data process 99分位时延 |
| |
get_sorted_shuffle_data_latency_sum |
s |
gRPC get_sorted_shuffle_data process 总时延 |
| gRPC Process时延数量 |
send_shuffle_data |
Count |
gRPC send_shuffle_data process 总数 |
| |
get_local_shuffle_data |
Count |
gRPC get_local_shuffle_data process 总数 |
| |
get_memory_shuffle_data |
Count |
gRPC get_memory_shuffle_data process 总数 |
| |
get_shuffle_result_for_multi_part |
Count |
gRPC get_shuffle_result_for_multi_part process总数 |
| |
report_shuffle_result |
Count |
gRPC report_shuffle_result process 总数 |
| |
get_sorted_shuffle_data |
Count |
gRPC get_sorted_shuffle_data process 总数 |
| gRPC请求数量 |
open |
Count |
gRPC open状态请求数 |
| |
send_shuffle_data_request_total |
Count |
netty send_shuffle_data_request总数 |
| |
get_local_shuffle_index_request_total |
Count |
netty get_local_shuffle_index_request 总数 |
| |
get_memory_shuffle_data_request_total |
Count |
netty get_memory_shuffle_data_request总数 |
| |
get_sorted_shuffle_data_request_total |
Count |
netty get_sorted_shuffle_data_request总数 |
| gRPC Transport时延 |
send_shuffle_data_latency_50 |
s |
grpc send_shuffle_data transport 50分位时延 |
| |
send_shuffle_data_latency_90 |
s |
grpc send_shuffle_data transport 90分位时延 |
| |
send_shuffle_data_latency_99 |
s |
grpc send_shuffle_data transport 99分位时延 |
| |
send_shuffle_data_latency_sum |
s |
grpc send_shuffle_data transport 总时延 |
| |
get_local_shuffle_data_latency_50 |
s |
grpc get_local_shuffle_data transport 50分位时延 |
| |
get_local_shuffle_data_latency_90 |
s |
grpc get_local_shuffle_data transport 90分位时延 |
| |
get_local_shuffle_data_latency_99 |
s |
grpc get_local_shuffle_data transport 99分位时延 |
| |
get_local_shuffle_data_latency_sum |
s |
grpc get_local_shuffle_data transport 总时延 |
| |
get_memory_shuffle_data_latency_50 |
s |
grpc get_memory_shuffle_data transport 50分位时延 |
| |
get_memory_shuffle_data_latency_90 |
s |
grpc get_memory_shuffle_data transport 90分位时延 |
| |
get_memory_shuffle_data_latency_99 |
s |
grpc get_memory_shuffle_data transport 99分位时延 |
| |
get_memory_shuffle_data_latency_sum |
s |
grpc get_memory_shuffle_data transport 总时延 |
| |
get_sorted_shuffle_data_latency_50 |
s |
grpc get_sorted_shuffle_data transport 50分位时延 |
| |
get_sorted_shuffle_data_latency_90 |
s |
grpc get_sorted_shuffle_data transport 90分位时延 |
| |
get_sorted_shuffle_data_latency_99 |
s |
grpc get_sorted_shuffle_data transport 99分位时延 |
| |
get_sorted_shuffle_data_latency_sum |
s |
grpc get_sorted_shuffle_data transport 总时延 |
| gRPC Transport时延数量 |
send_shuffle_data |
Count |
grpc send_shuffle_data transport 总数 |
| |
get_local_shuffle_data |
Count |
grpc get_local_shuffle_data transport 总数 |
| |
get_memory_shuffle_data |
Count |
grpc get_memory_shuffle_data transport 总数 |
| |
get_sorted_shuffle_data |
Count |
grpc get_sorted_shuffle_data transport 总数 |
| Netty连接 |
active_connection |
Count |
active netty连接数 |
| Netty异常 |
handle_exception |
Count |
netty handle异常数 |
| Netty Process时延 |
send_shuffle_data_request_latency_50 |
s |
netty send_shuffle_data_request process 50分位时延 |
| |
send_shuffle_data_request_latency_90 |
s |
netty send_shuffle_data_request process 90分位时延 |
| |
send_shuffle_data_request_latency_99 |
s |
netty send_shuffle_data_request process 99分位时延 |
| |
send_shuffle_data_request_latency_sum |
s |
netty send_shuffle_data_request process 总时延 |
| |
get_local_shuffle_data_request_latency_50 |
s |
netty get_local_shuffle_data_request process 50分位时延 |
| |
get_local_shuffle_data_request_latency_90 |
s |
netty get_local_shuffle_data_request process 90分位时延 |
| |
get_local_shuffle_data_request_latency_99 |
s |
netty get_local_shuffle_data_request process 99分位时延 |
| |
get_local_shuffle_data_request_latency_sum |
s |
netty get_local_shuffle_data_request process 总时延 |
| |
get_local_shuffle_index_request_latency_50 |
s |
netty get_local_shuffle_index_request process 50分位时延 |
| |
get_local_shuffle_index_request_latency_90 |
s |
netty get_local_shuffle_index_request process 90分位时延 |
| |
get_local_shuffle_index_request_latency_99 |
s |
netty get_local_shuffle_index_request process 99分位时延 |
| |
get_local_shuffle_index_request_latency_sum |
s |
netty get_local_shuffle_index_request process 总时延 |
| |
get_memory_shuffle_data_request_latency_50 |
s |
netty get_memory_shuffle_data_request process 50分位时延 |
| |
get_memory_shuffle_data_request_latency_90 |
s |
netty get_memory_shuffle_data_request process 90分位时延 |
| |
get_memory_shuffle_data_request_latency_99 |
s |
netty get_memory_shuffle_data_request process 99分位时延 |
| |
get_memory_shuffle_data_request_latency_sum |
s |
netty get_memory_shuffle_data_request process 总时延 |
| |
get_sorted_shuffle_data_request_latency_50 |
s |
netty get_sorted_shuffle_data_request process 50分位时延 |
| |
get_sorted_shuffle_data_request_latency_90 |
s |
netty get_sorted_shuffle_data_request process 90分位时延 |
| |
get_sorted_shuffle_data_request_latency_99 |
s |
netty get_sorted_shuffle_data_request process 99分位时延 |
| |
get_sorted_shuffle_data_request_latency_sum |
s |
netty get_sorted_shuffle_data_request process 总时延 |
| Netty Process时延数量 |
send_shuffle_data_request |
Count |
netty send_shuffle_data_request process 总数 |
| |
get_local_shuffle_data_request |
Count |
netty get_local_shuffle_data_request process 个数 |
| |
get_local_shuffle_index_request |
Count |
netty get_local_shuffle_index_request process 总数 |
| |
get_memory_shuffle_data_request |
Count |
netty get_memory_shuffle_data_request process 总数 |
| |
get_sorted_shuffle_data_request |
Count |
netty get_sorted_shuffle_data_request process 总数 |
| Netty Transport时延 |
send_shuffle_data_request_latency_50 |
s |
netty send_shuffle_data_request transport 50分位时延 |
| |
send_shuffle_data_request_latency_90 |
s |
netty send_shuffle_data_request transport 90分位时延 |
| |
send_shuffle_data_request_latency_99 |
s |
netty send_shuffle_data_request transport 99分位时延 |
| |
send_shuffle_data_request_latency_sum |
s |
netty send_shuffle_data_request transport 总时延 |
| |
get_local_shuffle_data_request_latency_50 |
s |
netty get_local_shuffle_data_request transport 50分位时延 |
| |
get_local_shuffle_data_request_latency_90 |
s |
netty get_local_shuffle_data_request transport 90分位时延 |
| |
get_local_shuffle_data_request_latency_99 |
s |
netty get_local_shuffle_data_request transport 99分位时延 |
| |
get_local_shuffle_data_request_latency_sum |
s |
netty get_local_shuffle_data_request transport 总时延 |
| |
get_local_shuffle_index_request_latency_50 |
s |
netty get_local_shuffle_index_request transport 50分位时延 |
| |
get_local_shuffle_index_request_latency_90 |
s |
netty get_local_shuffle_index_request transport 90分位时延 |
| |
get_local_shuffle_index_request_latency_99 |
s |
netty get_local_shuffle_index_request transport99分位时延 |
| |
get_local_shuffle_index_request_latency_sum |
s |
netty get_local_shuffle_index_request transport 总时延 |
| |
get_memory_shuffle_data_request_latency_50 |
s |
netty get_memory_shuffle_data_request transport 50分位时延 |
| |
get_memory_shuffle_data_request_latency_90 |
s |
netty get_memory_shuffle_data_request transport 90分位时延 |
| |
get_memory_shuffle_data_request_latency_99 |
s |
netty get_memory_shuffle_data_request transport 99分位时延 |
| |
get_memory_shuffle_data_request_latency_sum |
s |
netty get_memory_shuffle_data_request transport 总时延 |
| |
get_sorted_shuffle_data_request_latency_50 |
s |
netty get_sorted_shuffle_data_request transport 50分位时延 |
| |
get_sorted_shuffle_data_request_latency_90 |
s |
netty get_sorted_shuffle_data_request transport 90分位时延 |
| |
get_sorted_shuffle_data_request_latency_99 |
s |
netty get_sorted_shuffle_data_request transport 99分位时延 |
| |
get_sorted_shuffle_data_request_latency_sum |
s |
netty get_sorted_shuffle_data_request transport 总时延 |
| Netty Transport时延数量 |
send_shuffle_data_request |
Count |
netty send_shuffle_data_request transport 总数 |
| |
get_local_shuffle_data_request |
Count |
netty get_local_shuffle_data_request transport 总数 |
| |
get_local_shuffle_index_request |
Count |
netty get_local_shuffle_index_request transport 个数 |
| |
get_memory_shuffle_data_request |
Count |
netty get_memory_shuffle_data_request transport 总数 |
| |
get_sorted_shuffle_data_request |
Count |
netty get_sorted_shuffle_data_request transport 总数 |
| 健康状态 |
is_healthy |
Count |
当前节点是否健康 |
| Partition数量 |
partition_num_with_node |
Count |
当前在ShuffleServer上存储数据的 partition 的数量 |
| |
total_partition_num |
Count |
节点上存储 partition的数量 |
| |
total_huge_partition_num |
Count |
节点上超大partition的总数量 |
| |
huge_partition_num |
Count |
节点上当前超大 partition的数量 |
| Pending任务 |
pending_tasks_num_for_boss_group |
Count |
netty boss group pending任务数 |
| |
pending_tasks_num_for_worker_group |
Count |
netty worker group pending任务数 |
| Read数据量 |
total_read_data |
Bytes |
所有 application从 ShuffleServer 中读取的数据量,包括data和index 数据 |
| |
total_read_local_data_file |
Bytes |
ShuffleServer 中读取的数据量 |
| |
total_read_local_index_file |
Bytes |
ShuffleServer 中读取的index数据量 |
| |
total_read_memory_data |
Bytes |
ShuffleServer 从内存中读取的数据量 |
| |
read_local_data_file_buffer_size |
Bytes |
当前读本地存储数据大小 |
| |
read_local_index_file_buffer_size |
Bytes |
当前读本地存储索引大小 |
| |
read_memory_data_buffer_size |
Bytes |
当前读内存数据大小 |
| Remove App资源数量 |
total_time_count |
Count |
移除APP资源总次数 |
| |
total_by_shuffle_ids_time_count |
Count |
移除APP部分shuffleIds资源总次数 |
| Remove App资源耗时 |
total_time_50 |
s |
移除APP资源50分位耗时 |
| |
total_time_90 |
s |
移除APP资源90分位耗时 |
| |
total_time_99 |
s |
移除APP资源99分位耗时 |
| |
total_time_sum |
s |
移除APP资源总耗时 |
| |
total_by_shuffle_ids_time_50 |
s |
移除APP部分shuffleIds资源50分位耗时 |
| |
total_by_shuffle_ids_time_90 |
s |
移除APP部分shuffleIds资源90分位耗时 |
| |
total_by_shuffle_ids_time_99 |
s |
移除APP部分shuffleIds资源99分位耗时 |
| |
total_by_shuffle_ids_time_sum |
s |
移除APP部分shuffleIds资源总耗时 |
| 申请内存 |
total_require_read_memory_num |
Count |
读取数据申请的内存总次数 |
| |
total_require_read_memory_retry_num |
Count |
读取数据申请时进行重试申请的总次数 |
| |
total_require_read_memory_failed_num |
Count |
读取数据申请内存失败的总次数 |
| Buffer申请 |
total_failed_for_huge_partition |
Count |
shuffle client向shuffle server申请超大内存失败的数量,超大内存由 rss.server.huge-partition.size.threshold 定义 |
| |
total_failed_for_regular_partition |
Count |
shuffle client向 shuffle server申请内存存放shuffle 数据失败的数量 |
| 存储 |
storage_used_bytes |
Bytes |
存储占用大小 |
| |
local_storage_total_space |
Bytes |
当前节点本地存储总大小 |
| |
local_storage_whole_disk_used_space |
Bytes |
当前节点本地存储空间已使用大小 |
| |
local_storage_whole_disk_free_space |
Bytes |
当前节点本地存储空间空闲大小 |
| |
local_storage_service_used_space |
Bytes |
当前节点本地存储空间被本服务存储使用大小 |
| 存储目录 |
local_storage_total_dirs_num |
Count |
当前节点可用本地存储根目录数量 |
| |
local_storage_corrupted_dirs_num |
Count |
当前节点不可用本地存储根目录数量 |
| 存储比例 |
local_storage_used_space_ratio |
% |
当前节点本地存储空间使用比例 |
| App 数量 |
total_app_not_found_num |
Count |
appNotFound个数 |
| |
total_app_num |
Count |
节点上处理过所有application的数量 |
| |
total_app_with_huge_partition_num |
Count |
有超大 partition的application总数量 |
| |
app_with_huge_partition_num |
Count |
节点上拥有超大 partition的application的累加值 |
| gRPC线程池 |
blocking_queue_size |
Count |
grpc 线程池队列排队任务数 |
| |
server_connection_number |
Count |
grpc当前连接数 |
| Write Block数量 |
total_write_block |
Count |
ShuffleServer Flush 到磁盘/远端存储的Block数量 |
| Write次数 |
handler_write_time_summary_count |
Count |
ShuffleServer Flush 数据到磁盘/远端存储次数 |
| Write异常 |
total_write_exception |
Count |
Flush数据到Local/远程失败次数 |
| Write线程 |
total_write_handler |
Count |
ShuffleServer 并发 Flush 数据线程数量 |
| 慢Write |
total_write_slow |
Count |
Flush数据到Local/远程慢的次数(阈值由rss.server.write.slow.threshold控制) |
| Write耗时 |
total_write_time |
ms |
ShuffleServer Flush 数据到磁盘/远端存储的耗时 |
| |
handler_write_time_summary_50 |
ms |
ShuffleServer Flush 数据到磁盘/远端存储50分位耗时 |
| |
handler_write_time_summary_90 |
ms |
ShuffleServer Flush 数据到磁盘/远端存储90分位耗时 |
| |
handler_write_time_summary_99 |
ms |
ShuffleServer Flush 数据到磁盘/远端存储99分位耗时 |
| |
handler_write_time_summary_sum |
ms |
ShuffleServer Flush 数据到磁盘/远端存储总耗时 |
| |
total_write_lock_duration |
ms |
LocalFileWrite锁总耗时 |
| Process Cpu Load |
ProcessCpuLoad |
% |
CPU利用率 |
| GC count |
YGC |
Count |
Young GC 次数 |
| |
FGC |
Count |
Full GC 次数 |
| GC time |
FGCT |
s |
Full GC 消耗时间 |
| |
GCT |
s |
垃圾回收时间消耗 |
| |
YGCT |
s |
Young GC 消耗时间 |
| Memory area percentage |
S0 |
% |
Survivor 0区内存使用占比 |
| |
E |
% |
Eden 区内存使用占比 |
| |
CCS |
% |
Compressed class space 区内存使用占比 |
| |
S1 |
% |
Survivor 1区内存使用占比 |
| |
O |
% |
Old 区内存使用占比 |
| |
M |
% |
Metaspace 区内存使用占比 |
| JVM Memory |
MemHeapUsedM |
MBytes |
已使用的堆内存 |
| |
MemNonHeapUsedM |
MBytes |
已使用的堆外内存 |
| |
MemHeapCommittedM |
MBytes |
已提交的堆内存 |
| |
MemNonHeapCommittedM |
MBytes |
已提交的堆外内存 |
| |
MemHeapMaxM |
MBytes |
最大堆内存 |
| |
MemHeapInitM |
MBytes |
初始堆内存 |
| |
MemNonHeapInitM |
MBytes |
初始堆外内存 |
| File Descriptors |
MaxFileDescriptorCount |
Count |
最大文件描述符数 |
| |
OpenFileDescriptorCount |
Count |
打开文件描述符数 |
| JVM Threads |
PeakThreadCount |
Count |
峰值线程数 |
| |
ThreadCount |
Count |
总线程数量 |
| |
DaemonThreadCount |
Count |
Daemon线程数量 |