You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-23 - 06:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Mon Mar 23, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1933528876,8sleep20-21swapper/219:07:322
1933528876,8sleep20-21swapper/219:07:312
1922628875,9sleep00-21swapper/019:06:060
1922628875,9sleep00-21swapper/019:06:060
1925427966,9sleep30-21swapper/319:06:293
1925427966,9sleep30-21swapper/319:06:283
1938627861,13sleep10-21swapper/119:08:111
1938627861,13sleep10-21swapper/119:08:111
19622992318,1cyclictest13382-21kworker/3:121:13:523
19622992121,0cyclictest0-21swapper/322:20:143
1962099210,20cyclictest0-21swapper/200:17:152
1961799210,1cyclictest0-21swapper/122:20:161
1962099203,10cyclictest537-21snmpd21:35:152
19617992019,1cyclictest0-21swapper/123:42:201
19615992019,1cyclictest0-21swapper/022:53:510
19615992019,1cyclictest0-21swapper/022:53:500
1962099193,2cyclictest31603-21ntp_states21:40:212
19617991919,0cyclictest0-21swapper/123:01:061
1961599198,6cyclictest15626-21perf22:30:000
1962299180,11cyclictest0-21swapper/322:17:213
1961599187,6cyclictest31356-21munin-run22:20:000
19615991812,4cyclictest14031-21apache221:51:070
19620991716,1cyclictest32328-21strings22:00:202
19620991712,3cyclictest23936-21ls21:55:192
1961799172,5cyclictest0-21swapper/120:15:141
1961799172,15cyclictest25629-21latency_hist23:15:011
1961799171,4cyclictest0-21swapper/121:46:071
1961799170,1cyclictest0-21swapper/121:31:231
1961799170,17cyclictest0-21swapper/121:51:561
1961799170,15cyclictest0-21swapper/100:00:081
1961599171,2cyclictest0-21swapper/000:09:260
1961599170,10cyclictest0-21swapper/023:00:170
123782170,0sleep10-21swapper/123:06:171
40522160,0sleep20-21swapper/200:38:352
1962299161,15cyclictest0-21swapper/323:38:593
1962299161,14cyclictest0-21swapper/321:16:343
1962299160,2cyclictest141rcu_preempt21:36:253
1962299160,0cyclictest0-21swapper/321:54:153
1962299160,0cyclictest0-21swapper/300:02:263
1962099163,2cyclictest141rcu_preempt22:35:192
1962099160,2cyclictest0-21swapper/200:21:402
1961799169,2cyclictest7214-21grep22:05:141
1961799169,1cyclictest0-21swapper/122:29:541
1961799162,12cyclictest49-21kcompactd000:29:291
19617991614,1cyclictest0-21swapper/123:19:331
1961799160,0cyclictest0-21swapper/122:02:011
1961599160,11cyclictest0-21swapper/021:35:120
155332160,2sleep10-21swapper/123:47:051
1962299153,10cyclictest17696-21ssh23:10:113
1962299151,9cyclictest0-21swapper/319:55:553
19622991514,1cyclictest0-21swapper/322:55:353
1962299150,14cyclictest0-21swapper/323:16:063
19620991511,2cyclictest26217-21pmu-power22:55:192
1962099150,15cyclictest0-21swapper/223:32:172
19617991514,1cyclictest38150irq/131-snd_hda00:17:151
1961799150,14cyclictest0-21swapper/100:36:561
1961599150,10cyclictest0-21swapper/023:45:170
1962299146,4cyclictest225-21systemd-journal20:20:003
19622991410,2cyclictest39-21ksoftirqd/322:30:433
1962299140,13cyclictest0-21swapper/322:11:103
1962099148,5cyclictest14015-21apache222:08:332
1962099146,4cyclictest55550irq/126-enp2s0-00:04:292
19620991413,1cyclictest0-21swapper/222:29:072
19620991410,3cyclictest14424-21awk21:10:242
19620991410,2cyclictest28443-21tr20:35:142
1962099140,13cyclictest0-21swapper/223:06:592
19617991412,1cyclictest25544-21fschecks_time22:55:141
19617991412,1cyclictest0-21swapper/119:25:141
1961799140,1cyclictest0-21swapper/122:11:451
1961599143,10cyclictest6929-21apache_accesses22:05:110
1962299137,5cyclictest14015-21apache222:03:583
19622991312,1cyclictest0-21swapper/321:25:163
19622991310,2cyclictest7815-21tr19:50:213
19622991310,2cyclictest29876-21wc19:30:153
19622991310,2cyclictest28718-21ls19:25:263
19622991310,2cyclictest2302-21grep23:20:143
19622991310,2cyclictest21722-21cut00:10:163
19622991310,2cyclictest12157-21cron21:09:593
1962099139,3cyclictest7387-21cron20:59:592
1962099139,2cyclictest30225-21grep19:30:162
1962099138,3cyclictest32-21ksoftirqd/223:10:112
19620991310,2cyclictest3386-21ls23:20:212
19620991310,2cyclictest25958-21tr19:20:222
19620991310,2cyclictest21352-21wc21:15:142
19620991310,2cyclictest20308-21ssh23:50:152
19620991310,2cyclictest14269-21grep00:05:252
19620991310,2cyclictest10881-21cron23:45:002
1962099130,11cyclictest0-21swapper/222:44:182
1961799131,7cyclictest31523-21needreboot21:40:201
19617991312,1cyclictest0-21swapper/122:30:291
19617991310,2cyclictest22820-21grep00:30:171
1961599139,3cyclictest3419-21cron00:00:000
1961599138,4cyclictest30096-21cut00:15:150
1961599134,2cyclictest225-21systemd-journal00:11:440
19615991310,2cyclictest8288-21sed19:50:270
19615991310,2cyclictest31871-21head22:00:170
19615991310,2cyclictest24412-21cat21:55:260
19615991310,2cyclictest2313-21grep22:40:270
19615991310,2cyclictest20076-21tr19:10:170
1961599130,0cyclictest0-21swapper/021:45:220
1962299129,2cyclictest8568-21cron23:05:003
1962299129,2cyclictest6966-21ssh22:05:123
1962299129,2cyclictest6033-21sed20:55:173
1962299129,2cyclictest4585-21tr19:45:153
1962299129,2cyclictest3732-21ssh23:40:153
1962299129,2cyclictest30388-21grep21:20:183
1962299129,2cyclictest2686-21head19:40:163
1962299129,2cyclictest22105-21sort20:20:193
1962299129,2cyclictest18250-21cron23:30:003
1962299129,2cyclictest14605-21cut21:30:173
1962299129,2cyclictest10586-21grep22:45:263
1962299128,3cyclictest10825-21ssh21:47:363
1962299126,5cyclictest14031-21apache222:53:393
1962299126,5cyclictest14031-21apache222:53:393
19622991210,2cyclictest39-21ksoftirqd/322:41:053
19622991210,1cyclictest32683-21grep19:35:173
19622991210,1cyclictest32230-21grep21:40:263
1962099129,2cyclictest6712-21grep19:50:142
1962099129,2cyclictest5066-21head19:45:172
1962099129,2cyclictest31433-21ssh21:20:272
1962099129,2cyclictest27547-21wc23:15:202
1962099129,2cyclictest2608-21ssh23:00:262
1962099129,2cyclictest22232-21ntpq00:10:202
1962099129,2cyclictest21040-21ls19:10:232
1962099129,2cyclictest15207-21cut21:50:182
1962099129,2cyclictest14573-21head00:25:192
1962099129,2cyclictest12985-21cat20:00:272
1962099128,3cyclictest24544-21sh19:20:012
1962099128,3cyclictest23046-21sleep22:15:002
1962099128,3cyclictest22375-21cut00:30:152
1962099125,3cyclictest17718-21if_enp2s022:50:182
1962099125,3cyclictest17718-21if_enp2s022:50:172
1962099121,6cyclictest0-21swapper/222:17:212
1962099121,10cyclictest0-21swapper/220:15:182
1961799129,2cyclictest9805-21cron23:25:011
1961799129,2cyclictest7396-21grep19:50:171
1961799129,2cyclictest4622-21grep20:50:281
1961799129,2cyclictest30288-21cron20:40:001
1961799129,2cyclictest23863-21sed21:35:251
1961799129,2cyclictest23589-21grep21:55:171
1961799129,2cyclictest11162-21grep23:25:171
1961799121,2cyclictest12167-21diskmemload00:10:351
19617991210,2cyclictest4236-21kworker/1:221:25:221
19617991210,1cyclictest6120-21grep00:20:171
1961799120,2cyclictest0-21swapper/122:45:011
1961799120,1cyclictest0-21swapper/119:45:211
1961799120,11cyclictest29877-21grep21:20:151
1961799120,11cyclictest25121-21tr19:20:161
1961599129,2cyclictest6301-21cut21:25:160
1961599129,2cyclictest6040-21sed20:55:170
1961599129,2cyclictest5776-21cat00:00:270
1961599129,2cyclictest31070-21grep21:40:170
1961599129,2cyclictest30287-21head19:30:170
1961599129,2cyclictest20555-21head23:50:160
1961599129,2cyclictest16329-21cron22:50:000
1961599129,2cyclictest15109-21grep20:05:230
1961599129,2cyclictest12227-21cut20:00:170
1961599129,2cyclictest10609-21grep23:25:160
1961599127,3cyclictest31092-21gzip21:20:220
1961599124,5cyclictest1634-21cron19:40:010
1961599122,6cyclictest12167-21diskmemload23:35:130
1962299119,1cyclictest25674-21tr22:35:193
1962299118,2cyclictest9505-21chrt21:01:283
1962299118,2cyclictest8994-21cat23:05:013
1962299118,2cyclictest8022-21wc22:25:153
1962299118,2cyclictest5088-21sh00:20:013
1962299118,2cyclictest32394-21chrt20:41:163
1962299118,2cyclictest31162-21sh23:56:213
1962299118,2cyclictest30296-21sh20:40:003
1962299118,2cyclictest30290-21sh00:35:063
1962299118,2cyclictest29610-21tr00:15:143
1962299118,2cyclictest25423-21chrt20:28:313
1962299118,2cyclictest24343-21sh21:55:233
1962299118,2cyclictest22477-21ls00:30:153
1962299118,2cyclictest20193-21expr23:50:153
1962299118,2cyclictest16272-21cut20:10:133
1962299118,2cyclictest13787-21cut00:25:163
1962299118,2cyclictest13549-21cron20:05:003
1962299118,2cyclictest12597-21ssh23:45:203
1962299113,3cyclictest0-21swapper/323:30:133
1962299110,10cyclictest12712-21expr00:05:143
1962099118,2cyclictest9286-21tr19:55:142
1962099118,2cyclictest8810-21ntpq21:00:222
1962099118,2cyclictest8152-21ssh21:45:282
1962099118,2cyclictest5765-21cut21:25:142
1962099118,2cyclictest29025-21ssh23:55:202
1962099118,2cyclictest27133-21apache_accesses23:35:102
1962099118,2cyclictest23995-21grep20:25:172
1962099118,2cyclictest23386-21cat20:25:002
1962099118,2cyclictest17565-21ssh21:31:462
1962099118,2cyclictest16271-21grep22:30:122
1962099118,2cyclictest11410-21cut23:25:202
1962099118,2cyclictest11152-21ssh23:45:032
1962099115,2cyclictest1994-21aten2_r7power_e19:40:142
1962099114,5cyclictest15277-21python20:05:252
1962099110,1cyclictest0-21swapper/222:46:232
1961799119,1cyclictest6898-21grep20:55:251
1961799119,1cyclictest26320-21taskset22:35:251
1961799119,1cyclictest25082-21grep20:25:281
1961799119,1cyclictest2508-21rm23:59:181
1961799118,2cyclictest9795-21turbostat.cron21:05:001
1961799118,2cyclictest5890-21ssh22:43:141
1961799118,2cyclictest28318-21cat23:35:181
1961799118,2cyclictest24021-21expr22:15:151
1961799118,2cyclictest19785-21ssh23:50:121
1961799118,2cyclictest19122-21cut23:30:151
1961799118,2cyclictest15388-21cat20:05:271
1961799118,2cyclictest13110-21grep00:05:161
1961799118,2cyclictest1279-21chrt19:35:321
1961599119,1cyclictest19106-21tr23:30:150
1961599118,2cyclictest9799-21turbostat21:05:000
1961599118,2cyclictest9433-21cat19:55:150
1961599118,2cyclictest5628-21grep00:20:140
1961599118,2cyclictest5484-21ssh23:21:340
1961599118,2cyclictest4837-21taskset00:39:100
1961599118,2cyclictest465-21cut22:20:180
1961599118,2cyclictest2874-21ssh23:40:020
1961599118,2cyclictest27469-21expr19:25:150
1961599118,2cyclictest26468-21expr23:15:110
1961599118,2cyclictest24195-21ssh22:35:020
1961599118,2cyclictest23449-21grep00:30:220
1961599118,2cyclictest20784-21ssh21:15:030
1961599118,2cyclictest20705-21chrt20:19:060
1961599118,2cyclictest18534-21ssh22:30:450
1961599118,2cyclictest17989-21tr23:10:130
1961599118,2cyclictest17684-21cat20:10:250
1961599118,2cyclictest15463-21ssh21:30:240
1961599118,2cyclictest15224-21rm22:10:120
1961599118,2cyclictest15154-21ssh21:10:460
1961599118,2cyclictest13729-21rm00:25:130
1961599112,6cyclictest49-21kcompactd019:48:550
1961599110,4cyclictest141rcu_preempt20:45:000
1962299108,1cyclictest217602sleep319:12:423
1962299107,2cyclictest26757-21chrt19:22:473
1962299107,2cyclictest26097-21tr20:30:153
1962299107,2cyclictest23271-21basename19:15:183
1962299101,5cyclictest3891-21cat20:50:203
1962299101,2cyclictest0-21swapper/320:05:133
1962099108,1cyclictest31595-21cat20:40:192
1962099108,1cyclictest3122-21cat20:50:162
1962099108,1cyclictest26026-21cat20:30:142
1962099108,1cyclictest18103-21taskset20:11:492
1962099108,1cyclictest1465-21taskset19:37:542
1962099107,2cyclictest666-21tr20:45:142
1962099107,2cyclictest31657-21ssh22:20:042
1961799108,2cyclictest13683-21cat20:05:001
1961799108,1cyclictest31047-21expr20:40:151
1961799108,1cyclictest2648-21sh20:50:001
1961799108,1cyclictest26021-21cat20:30:151
1961799108,1cyclictest21491-21expr20:20:141
1961799108,1cyclictest20303-21expr19:10:171
1961799108,1cyclictest18882sleep119:40:041
1961799108,1cyclictest18832-21rm22:50:301
1961799108,1cyclictest18832-21rm22:50:291
1961799108,1cyclictest16304-21hostname20:10:141
1961799108,1cyclictest10734-21sed19:55:261
1961799107,2cyclictest31250-21tail19:30:251
1961799107,2cyclictest21851-21head21:15:161
1961799107,2cyclictest12927-21tr21:10:141
1961799107,2cyclictest12139-21taskset21:09:511
1961599108,1cyclictest8817-21date23:05:000
1961599108,1cyclictest26795-21cat20:30:200
1961599108,1cyclictest25533-21cat20:29:550
1961599107,2cyclictest2871-21chrt20:50:030
1961599106,2cyclictest2637-21head19:40:170
1961599106,2cyclictest0-21swapper/019:20:270
1961599100,9cyclictest24774-21ssh22:55:090
1961599100,6cyclictest225-21systemd-journal20:49:590
196229997,1cyclictest24012sleep320:46:583
196209992,3cyclictest225-21systemd-journal19:26:552
196179997,1cyclictest22979-21df_abs19:15:161
196159997,1cyclictest242532sleep019:16:230
196159996,2cyclictest22161-21tail20:20:180
196159992,3cyclictest225-21systemd-journal20:40:000
196159990,3cyclictest11991-21sshd21:07:550
196209985,2cyclictest10308-21cut21:05:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional