You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-07 - 03:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot6s.osadl.org (updated Fri Nov 07, 2025 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
24589211996,18sleep30-21swapper/319:07:173
24579211997,16sleep00-21swapper/019:07:100
246752115104,6sleep20-21swapper/219:08:252
24674210992,12sleep10-21swapper/119:08:241
282612720,1sleep10-21swapper/123:09:101
207712720,1sleep20-21swapper/200:07:112
24829993717,18cyclictest0-21swapper/220:10:112
24830993313,18cyclictest0-21swapper/321:30:123
193732250,0sleep20-21swapper/221:53:432
142352240,0sleep30-21swapper/322:20:273
246132230,0sleep30-21swapper/323:38:193
2482999220,20cyclictest0-21swapper/220:00:132
2482799220,20cyclictest0-21swapper/023:00:110
2483099210,3cyclictest30290-21ssh22:05:113
2483099210,3cyclictest0-21swapper/323:15:183
2483099210,3cyclictest0-21swapper/320:00:273
2483099200,3cyclictest5464-21munin-node3
2483099200,3cyclictest3162-21ssh22:10:153
2483099200,3cyclictest2746-21ssh00:21:593
2483099200,3cyclictest24519-21ssh22:31:193
2483099200,3cyclictest19295-21ssh23:00:033
2483099200,3cyclictest18270-21sh00:38:073
2483099200,3cyclictest10591-21ssh22:17:233
2483099200,3cyclictest0-21swapper/323:05:273
2483099200,3cyclictest0-21swapper/320:20:273
2483099200,18cyclictest3853-21ssh22:44:263
2483099200,18cyclictest0-21swapper/323:46:333
2483099200,18cyclictest0-21swapper/322:00:153
2483099200,12cyclictest0-21swapper/321:58:153
2483099190,3cyclictest15309-21ssh21:49:593
2483099190,3cyclictest13705-21grep00:00:193
2483099190,3cyclictest11680-21ssh23:58:113
2483099190,2cyclictest9475-21fschecks_count21:10:153
2483099190,2cyclictest6875-21diskmemload00:25:353
2483099190,2cyclictest4876-21ssh21:39:063
2483099190,2cyclictest29235-21rm22:36:033
2483099190,2cyclictest20464-21ssh21:20:393
2483099190,17cyclictest0-21swapper/323:33:023
2483099190,17cyclictest0-21swapper/323:12:163
2482999190,17cyclictest0-21swapper/222:40:002
2483099180,3cyclictest18328-21ssh21:52:073
2483099180,2cyclictest144002sleep300:34:393
2483099180,2cyclictest0-21swapper/300:12:113
2483099180,11cyclictest0-21swapper/300:16:013
24829991813,3cyclictest29121-21ssh23:10:122
2483099170,3cyclictest6222-21apt-get23:20:073
2483099170,3cyclictest5010-21snmpd19:26:073
2483099170,3cyclictest21836-21ssh22:29:383
2483099170,3cyclictest0-21swapper/320:06:173
2483099170,2cyclictest8697-21diskmemload22:50:043
2483099170,2cyclictest5340-21ssh23:51:023
2483099170,16cyclictest0-21swapper/300:05:543
2483099170,15cyclictest0-21swapper/323:26:103
2482999170,5cyclictest0-21swapper/221:12:382
2482899170,15cyclictest0-21swapper/122:16:491
2483099160,10cyclictest0-21swapper/323:43:193
2483099160,10cyclictest0-21swapper/322:55:573
2482999161,3cyclictest10311-21iostat22:50:162
24829991611,3cyclictest4487-21latency_hist19:39:592
2482999160,14cyclictest0-21swapper/223:29:462
2482899166,7cyclictest5911-21gdm-simple-gree19:14:581
24828991612,2cyclictest0-21swapper/123:36:101
2482799161,3cyclictest0-21swapper/000:25:140
24827991611,3cyclictest18939-21ssh23:32:010
24827991611,3cyclictest0-21swapper/023:35:170
2482799160,14cyclictest14776-21latency_hist00:35:000
2483099155,8cyclictest8687-21cron21:10:003
2483099154,9cyclictest5089-21df21:00:133
2483099154,3cyclictest2-21kthreadd20:45:133
2483099153,3cyclictest5872-21cut19:40:223
2483099153,3cyclictest4560-21ntpd19:45:173
2483099153,3cyclictest26670-21ssh21:27:423
2483099153,3cyclictest26016-21processes19:10:203
2483099153,3cyclictest23268-21expr20:25:153
2483099153,3cyclictest20323-21grep20:15:293
2483099153,3cyclictest10616-21sed21:45:013
2483099152,3cyclictest30456-21grep19:20:273
2483099151,3cyclictest0-21swapper/319:35:153
2482999152,11cyclictest16464-1kworker/u9:223:01:472
24829991510,3cyclictest0-21swapper/200:13:112
24828991512,2cyclictest0-21swapper/121:21:501
24828991512,2cyclictest0-21swapper/100:38:501
24828991510,3cyclictest21615-21cut21:55:211
2482899150,3cyclictest0-21swapper/100:18:221
2482899150,13cyclictest0-21swapper/121:10:121
24827991512,2cyclictest25272-21kworker/u8:221:34:050
2482799151,12cyclictest0-21swapper/022:33:390
24827991510,3cyclictest0-21swapper/023:28:090
2482799150,13cyclictest4324-21nscd23:43:220
2482799150,13cyclictest0-21swapper/022:15:190
2482799150,13cyclictest0-21swapper/019:50:170
75552140,2sleep00-21swapper/021:40:310
2483099141,3cyclictest25317-21cpuspeed_turbos20:30:163
2483099141,3cyclictest0-21swapper/321:15:063
2483099141,3cyclictest0-21swapper/320:55:143
2483099141,3cyclictest0-21swapper/320:50:173
2483099141,3cyclictest0-21swapper/320:40:133
2483099141,3cyclictest0-21swapper/320:35:133
2483099141,3cyclictest0-21swapper/320:10:033
2483099141,3cyclictest0-21swapper/319:55:063
2483099141,3cyclictest0-21swapper/319:50:153
2483099141,3cyclictest0-21swapper/319:30:143
2483099141,3cyclictest0-21swapper/319:15:083
2482999142,6cyclictest0-21swapper/221:05:002
2482999142,6cyclictest0-21swapper/220:30:002
2482999142,6cyclictest0-21swapper/200:25:012
24829991411,2cyclictest0-21swapper/223:20:062
2482999140,3cyclictest9427-21ssh22:15:342
2482999140,3cyclictest16106-21ssh22:55:292
2482999140,13cyclictest3240-21ssh23:50:062
2482999140,13cyclictest0-21swapper/219:40:262
2482999140,13cyclictest0-21swapper/200:25:162
2482999140,12cyclictest18772-21ssh21:20:122
2482999140,12cyclictest0-21swapper/223:32:262
2482999140,12cyclictest0-21swapper/223:15:242
2482999140,12cyclictest0-21swapper/222:20:332
2482999140,12cyclictest0-21swapper/221:27:542
2482999140,12cyclictest0-21swapper/221:16:052
2482899149,3cyclictest32510-21chrt19:25:291
2482899142,7cyclictest16767-21copy20:10:001
2482899142,7cyclictest0-21swapper/100:05:001
2482899142,6cyclictest0-21swapper/120:55:011
24828991411,2cyclictest332250irq/24-eth123:16:421
24828991411,2cyclictest0-21swapper/123:23:001
24828991411,2cyclictest0-21swapper/122:42:291
24828991411,2cyclictest0-21swapper/100:13:301
2482899140,3cyclictest25990-21ps20:30:221
2482899140,2cyclictest0-21swapper/122:30:501
2482899140,2cyclictest0-21swapper/100:27:101
2482899140,13cyclictest0-21swapper/123:51:001
2482899140,13cyclictest0-21swapper/119:30:301
2482899140,13cyclictest0-21swapper/100:31:591
2482899140,12cyclictest0-21swapper/122:38:231
2482799142,7cyclictest6283-21copy23:20:000
2482799142,11cyclictest0-21swapper/023:52:060
24827991411,2cyclictest8911-21kworker/u8:121:47:050
24827991411,2cyclictest8697-21diskmemload22:14:220
2482799140,2cyclictest0-21swapper/021:16:410
2482799140,2cyclictest0-21swapper/020:35:010
2482799140,12cyclictest0-21swapper/023:13:270
2482799140,12cyclictest0-21swapper/022:50:000
2482799140,12cyclictest0-21swapper/021:37:590
275482130,2sleep20-21swapper/222:01:192
2482999138,3cyclictest0-21swapper/219:20:142
24829991310,2cyclictest0-21swapper/222:09:352
24829991310,2cyclictest0-21swapper/219:15:222
2482999130,2cyclictest26153-21python19:10:242
2482999130,2cyclictest0-21swapper/223:40:432
2482999130,2cyclictest0-21swapper/223:37:152
2482999130,2cyclictest0-21swapper/222:49:362
2482999130,2cyclictest0-21swapper/221:40:072
2482999130,12cyclictest0-21swapper/223:48:142
2482999130,12cyclictest0-21swapper/220:30:312
2482999130,11cyclictest0-21swapper/222:30:192
2482899139,2cyclictest0-21swapper/123:40:511
2482899131,7cyclictest0-21swapper/120:55:001
2482899131,10cyclictest0-21swapper/121:35:211
24828991310,2cyclictest0-21swapper/123:00:101
24828991310,2cyclictest0-21swapper/122:55:441
24828991310,2cyclictest0-21swapper/122:27:231
24828991310,2cyclictest0-21swapper/121:51:091
24828991310,2cyclictest0-21swapper/121:34:401
24828991310,2cyclictest0-21swapper/100:01:411
2482899130,2cyclictest9604-21apt-get22:50:121
2482899130,2cyclictest0-21swapper/123:29:441
2482899130,2cyclictest0-21swapper/122:20:241
2482899130,2cyclictest0-21swapper/121:40:221
2482899130,12cyclictest0-21swapper/123:49:001
2482899130,12cyclictest0-21swapper/123:32:091
2482899130,12cyclictest0-21swapper/122:49:201
2482899130,12cyclictest0-21swapper/122:00:001
2482899130,12cyclictest0-21swapper/121:15:191
2482899130,11cyclictest8697-21diskmemload21:45:301
2482799138,3cyclictest14524-21ssh22:55:130
2482799131,3cyclictest0-21swapper/000:36:450
24827991310,2cyclictest15022-21ssh22:21:390
24827991310,2cyclictest0-21swapper/023:45:140
24827991310,2cyclictest0-21swapper/021:56:020
24827991310,2cyclictest0-21swapper/021:25:020
24827991310,2cyclictest0-21swapper/000:21:110
2482799130,2cyclictest8697-21diskmemload00:05:160
2482799130,2cyclictest0-21swapper/022:51:200
2482799130,2cyclictest0-21swapper/020:40:310
2482799130,12cyclictest0-21swapper/022:28:220
2482799130,12cyclictest0-21swapper/021:54:420
2482799130,11cyclictest8697-21diskmemload22:00:220
2482799130,11cyclictest2092-21ssh22:41:420
2482799130,11cyclictest0-21swapper/023:05:210
2482799130,11cyclictest0-21swapper/019:35:270
2482999129,2cyclictest0-21swapper/222:26:422
2482999129,2cyclictest0-21swapper/221:48:162
2482999129,2cyclictest0-21swapper/220:20:282
2482999122,7cyclictest0-21swapper/219:58:022
2482999120,2cyclictest0-21swapper/222:40:542
2482999120,2cyclictest0-21swapper/220:55:292
2482999120,2cyclictest0-21swapper/200:15:342
2482999120,2cyclictest0-21swapper/200:02:312
2482999120,11cyclictest0-21swapper/223:55:342
2482999120,11cyclictest0-21swapper/223:07:252
2482999120,11cyclictest0-21swapper/222:10:142
2482999120,11cyclictest0-21swapper/221:55:072
2482999120,11cyclictest0-21swapper/221:36:052
2482999120,11cyclictest0-21swapper/221:30:042
2482999120,11cyclictest0-21swapper/221:05:232
2482999120,11cyclictest0-21swapper/220:40:232
2482999120,11cyclictest0-21swapper/200:39:282
2482999120,11cyclictest0-21swapper/200:30:032
2482899129,2cyclictest8697-21diskmemload22:13:251
2482899129,2cyclictest0-21swapper/123:11:001
2482899127,3cyclictest23021-21munin-node1
2482899127,3cyclictest0-21swapper/121:05:161
2482899122,8cyclictest21110-21aten2.4-expect20:20:141
2482899121,9cyclictest0-21swapper/119:50:001
2482899120,2cyclictest0-21swapper/100:21:541
2482899120,1cyclictest0-21swapper/123:59:411
2482899120,11cyclictest0-21swapper/121:25:411
2482899120,10cyclictest0-21swapper/122:05:211
2482799129,2cyclictest6573-21kworker/u8:121:20:050
2482799129,2cyclictest4550-21copy21:00:020
2482799129,2cyclictest0-21swapper/021:05:200
2482799129,2cyclictest0-21swapper/020:35:170
2482799128,2cyclictest9651-21smartctl21:10:160
2482799120,2cyclictest0-21swapper/022:05:290
2482799120,2cyclictest0-21swapper/020:25:160
2482799120,2cyclictest0-21swapper/019:15:210
2482799120,2cyclictest0-21swapper/000:10:150
2482799120,11cyclictest0-21swapper/023:55:410
2482799120,11cyclictest0-21swapper/022:35:570
2482799120,11cyclictest0-21swapper/020:20:170
2482799120,11cyclictest0-21swapper/020:10:190
2482799120,11cyclictest0-21swapper/000:16:400
2482799120,11cyclictest0-21swapper/000:00:190
2482799120,10cyclictest8697-21diskmemload23:20:250
2482999116,3cyclictest1491-21needreboot20:50:212
2482899118,2cyclictest0-21swapper/119:35:201
2482899117,2cyclictest0-21swapper/120:10:131
2482899111,8cyclictest8882-21apt-get19:50:131
2482899111,8cyclictest0-21swapper/119:40:171
2482899111,3cyclictest0-21swapper/119:20:251
2482899110,9cyclictest0-21swapper/120:35:151
2482899110,3cyclictest0-21swapper/120:15:211
2482899110,2cyclictest0-21swapper/119:55:231
2482799118,2cyclictest0-21swapper/019:30:220
2482799118,2cyclictest0-21swapper/019:25:160
2482799116,3cyclictest0-21swapper/020:55:200
2482799112,6cyclictest0-21swapper/019:11:590
2482799110,2cyclictest0-21swapper/019:40:230
2482799110,1cyclictest0-21swapper/019:55:230
2482999107,2cyclictest31098-21apt-get19:25:152
2482999105,3cyclictest8898-21aten2.4_r3power19:50:132
2482999100,9cyclictest0-21swapper/220:15:192
2482899105,3cyclictest30887-21ls20:45:001
2482899100,8cyclictest31088-21munin-node1
2482899100,8cyclictest0-21swapper/121:00:191
2482799107,2cyclictest0-21swapper/020:45:270
2482799105,3cyclictest875-21tr20:50:160
248299994,3cyclictest7582-21grep19:45:172
248299994,3cyclictest32407-21unixbench_multi20:45:272
248299991,6cyclictest0-21swapper/220:35:002
248289994,3cyclictest12919-21apt20:00:111
248289990,2cyclictest0-21swapper/119:15:131
248279994,3cyclictest5129-21runrttasks19:50:000
248279994,3cyclictest30874-21latency_hist19:25:000
248279994,3cyclictest19118-21aten2.4_r3power20:15:160
248279994,3cyclictest15772-21needreboot20:05:210
248279994,3cyclictest13334-21fschecks_count20:00:150
248299984,2cyclictest149122sleep220:05:022
248299980,6cyclictest622-21chrt19:30:032
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional