You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 12:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot7s.osadl.org (updated Fri May 02, 2025 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7780210187,9sleep10-21swapper/119:09:411
777229581,10sleep30-21swapper/319:09:343
748128371,8sleep00-21swapper/019:05:460
766828265,13sleep20-21swapper/219:08:122
791099290,2cyclictest0-21swapper/022:25:260
7912992423,1cyclictest0-21swapper/122:47:281
7912992423,1cyclictest0-21swapper/122:47:271
791599220,22cyclictest0-21swapper/200:00:412
7918992121,0cyclictest0-21swapper/322:41:583
7915992119,2cyclictest32-21ksoftirqd/223:52:102
791899201,1cyclictest0-21swapper/323:32:273
791599202,10cyclictest0-21swapper/221:40:162
791599200,20cyclictest0-21swapper/200:32:262
791599200,19cyclictest0-21swapper/220:55:182
791599190,17cyclictest0-21swapper/222:30:032
791099196,6cyclictest25939-21awk20:55:000
791099190,1cyclictest0-21swapper/022:50:160
7915991814,2cyclictest6420-21unixbench_singl21:35:302
791599181,17cyclictest0-21swapper/222:17:112
791599180,18cyclictest7091-21kworker/2:122:29:192
791599180,0cyclictest0-21swapper/222:51:392
791099180,17cyclictest0-21swapper/022:40:130
791099180,15cyclictest0-21swapper/021:17:370
791899170,17cyclictest0-21swapper/321:14:513
791599170,16cyclictest0-21swapper/223:59:482
791599170,14cyclictest0-21swapper/223:17:142
791299170,1cyclictest0-21swapper/123:35:221
127172170,0sleep30-21swapper/322:47:573
127172170,0sleep30-21swapper/322:47:563
7918991614,2cyclictest0-21swapper/321:16:133
7915991615,1cyclictest11535-21kworker/2:200:26:172
7915991614,2cyclictest0-21swapper/223:13:282
7915991613,2cyclictest0-21swapper/220:30:152
791099160,12cyclictest0-21swapper/019:30:140
791899158,4cyclictest1378-21ntpd22:18:233
791599158,6cyclictest7415-21perf23:30:002
791599153,10cyclictest0-21swapper/223:35:462
7915991514,1cyclictest17945-21kworker/2:121:30:592
791599150,15cyclictest0-21swapper/221:19:392
791299150,1cyclictest0-21swapper/122:10:541
7910991514,1cyclictest0-21swapper/000:17:560
791099150,1cyclictest0-21swapper/023:22:540
791099150,1cyclictest0-21swapper/021:30:430
7918991412,1cyclictest0-21swapper/321:25:393
7918991411,2cyclictest0-21swapper/323:41:013
7918991410,3cyclictest27387-21ls00:30:233
7918991410,3cyclictest22979-21tr19:40:223
791899140,2cyclictest0-21swapper/323:16:053
791599147,6cyclictest15243-21basename19:25:002
7915991411,2cyclictest295872sleep221:51:562
7915991410,2cyclictest24856-21cat19:45:002
791599140,9cyclictest0-21swapper/220:45:202
791599140,5cyclictest495-21cron20:45:002
791599140,14cyclictest0-21swapper/221:21:292
7912991414,0cyclictest0-21swapper/100:24:181
7910991413,1cyclictest0-21swapper/021:57:530
7910991410,3cyclictest5352-21sed21:35:210
791099140,13cyclictest19757-21sensors00:00:250
791899139,3cyclictest22052-21sh21:25:003
791899139,3cyclictest21187-21wc20:45:003
791899139,3cyclictest16134-21sort22:05:223
791899132,4cyclictest3673-21latency_hist23:50:003
7918991310,2cyclictest4196-21grep23:50:183
7918991310,2cyclictest31209-21grep21:05:163
7918991310,2cyclictest28966-21cat19:50:283
7918991310,2cyclictest20053-21ssh00:25:133
7918991310,2cyclictest16001-21grep23:35:183
7918991310,2cyclictest15758-21grep00:20:303
7918991310,2cyclictest12581-21wc23:10:243
7918991310,2cyclictest12478-21wc20:25:203
791599139,3cyclictest20036-21wc19:35:002
791599139,3cyclictest16263-21uname22:05:232
791599139,3cyclictest13918-21wc20:30:002
7915991310,2cyclictest626-21diskmemload23:30:502
7915991310,2cyclictest30527-21head19:55:202
791599130,1cyclictest0-21swapper/200:08:032
791599130,12cyclictest626-21diskmemload23:41:282
791299139,3cyclictest29516-21ssh23:45:171
791299139,3cyclictest21843-21grep00:25:271
791299134,7cyclictest7744-21snmpd00:08:071
7912991312,1cyclictest0-21swapper/100:37:461
7912991310,2cyclictest8103-21sort00:15:171
7912991310,2cyclictest4932-21grep23:50:201
7912991310,2cyclictest31818-21sh00:35:001
7912991310,2cyclictest29300-21grep21:00:231
7912991310,2cyclictest2079-21tr21:55:231
7912991310,2cyclictest18988-21grep00:00:181
7912991310,2cyclictest15731-21tr19:25:161
791299130,1cyclictest0-21swapper/123:28:331
791299130,12cyclictest29030-21grep19:50:291
791299130,12cyclictest1468-21grep21:10:181
791099139,3cyclictest16956-21ssh21:20:290
791099139,2cyclictest0-21swapper/021:10:170
791099136,1cyclictest0-21swapper/023:09:450
7910991310,2cyclictest9863-21grep22:00:280
7910991310,2cyclictest5569-21cron00:40:010
7910991310,2cyclictest3557-21cat20:05:290
7910991310,2cyclictest31911-21wc21:05:210
7910991310,2cyclictest29332-21tail21:00:230
7910991310,2cyclictest22676-21cut23:40:160
7910991310,2cyclictest14225-21cut19:20:200
791899129,2cyclictest9984-21grep20:20:203
791899129,2cyclictest8214-21sed00:15:173
791899129,2cyclictest5219-21grep20:10:203
791899129,2cyclictest31115-21tr21:30:273
791899129,2cyclictest29231-21sort21:00:223
791899129,2cyclictest29144-21cron22:15:003
791899129,2cyclictest29074-21cat23:00:013
791899129,2cyclictest26687-21grep22:35:173
791899129,2cyclictest25830-21grep19:45:223
791899129,2cyclictest25459-21ssh23:20:013
791899129,2cyclictest24046-21cut20:50:183
791899129,2cyclictest0-21swapper/322:27:133
791899128,3cyclictest4251-21grep20:09:593
791899125,6cyclictest39-21ksoftirqd/300:00:003
791899120,11cyclictest16249-21head19:25:183
791599129,3cyclictest0-21swapper/200:13:202
791599129,2cyclictest903-21grep21:55:152
791599129,2cyclictest28578-21seq21:00:182
791599129,2cyclictest27151-21cron19:49:592
791599129,2cyclictest26150-21chrt22:57:172
791599129,2cyclictest22692-21cut22:10:132
791599129,2cyclictest21898-21grep19:35:262
791599128,3cyclictest634-21tail00:35:192
791599127,4cyclictest9538-21ssh00:15:432
791599127,4cyclictest31640-21ssh23:45:432
791599127,4cyclictest26329-21cut22:35:162
7915991212,0cyclictest0-21swapper/219:50:172
7915991211,1cyclictest0-21swapper/221:13:052
7915991210,1cyclictest8774-21cut19:10:192
7915991210,1cyclictest8774-21cut19:10:182
791599120,3cyclictest0-21swapper/223:03:552
791299129,2cyclictest9665-21tr21:15:231
791299129,2cyclictest9418-21head22:00:241
791299129,2cyclictest7046-21wc00:15:001
791299129,2cyclictest5270-21grep21:35:211
791299129,2cyclictest32739-21cron21:54:591
791299129,2cyclictest30078-21cut19:55:181
791299129,2cyclictest12646-21wc23:10:241
791299129,2cyclictest12227-21wc23:55:201
791299129,2cyclictest12019-21grep21:40:181
791299128,3cyclictest31627-21cut21:05:181
791299128,3cyclictest30852-21grep22:15:201
791299128,3cyclictest22470-21grep23:40:151
791299127,3cyclictest0-21swapper/122:27:291
791099129,2cyclictest7984-21grep20:15:230
791099129,2cyclictest3615-21munin-plugin-st23:49:590
791099129,2cyclictest31989-21cat19:59:590
791099129,2cyclictest30688-21tr23:00:230
791099129,2cyclictest29899-21grep22:15:150
791099129,2cyclictest18592-21cat21:45:010
791099129,2cyclictest14830-21turbostat22:50:010
791099129,2cyclictest14830-21turbostat22:50:000
791099129,2cyclictest11475-21sh23:10:150
791099129,2cyclictest1117-21ssh00:10:200
791099129,2cyclictest10253-21sleep19:15:010
791099129,2cyclictest10253-21sleep19:15:000
791099128,2cyclictest22648-21tr22:10:140
791099126,5cyclictest23555-21munin-run20:50:000
791099120,7cyclictest22423-21perf19:40:000
791099120,11cyclictest9538-21tr20:20:190
61302120,1sleep30-21swapper/323:05:353
791899119,1cyclictest1819-21sensors00:10:253
791899118,2cyclictest945-21tr21:55:163
791899118,2cyclictest9079-21tr22:00:213
791899118,2cyclictest8860-21grep19:10:193
791899118,2cyclictest8860-21grep19:10:193
791899118,2cyclictest7791-21ssh22:21:503
791899118,2cyclictest6471-21ssh21:35:313
791899118,2cyclictest30442-21ssh23:00:203
791899118,2cyclictest28204-21chrt20:59:043
791899118,2cyclictest27728-21wc21:50:283
791899118,2cyclictest21042-21sh22:30:443
791899118,2cyclictest20313-21basename19:35:143
791899118,2cyclictest19309-21rm22:52:293
791899118,2cyclictest18783-21ssh21:45:103
791899118,2cyclictest18104-21expr19:30:153
791899118,2cyclictest18027-21ssh00:00:043
791899118,2cyclictest17073-21grep20:35:173
791899118,2cyclictest1680-21sshd20:02:403
791899118,2cyclictest16160-21sleep20:35:013
791899118,2cyclictest1589-21date00:35:283
791899118,2cyclictest13827-21head19:20:183
791899118,2cyclictest12852-21cat19:20:003
791899117,3cyclictest632-21grep23:25:163
791899113,7cyclictest11661-21/usr/sbin/munin21:40:153
791599118,2cyclictest6046-21ssh23:05:302
791599118,2cyclictest5909-21ssh22:43:182
791599118,2cyclictest5811-21grep22:20:252
791599118,2cyclictest27643-21sed23:20:282
791599118,2cyclictest15078-21grep00:20:232
791599118,2cyclictest10752-21sed20:20:292
791599118,2cyclictest10158-21ssh22:45:282
791599118,2cyclictest10158-21ssh22:45:282
791599117,3cyclictest25403-21cat20:50:292
791599117,3cyclictest15803-21wc19:25:152
791599117,2cyclictest4006-21taskset20:07:062
791599112,2cyclictest8761-21sh22:00:182
791599111,8cyclictest18088-21head20:35:262
791599110,10cyclictest6345-21chrt20:11:392
791299119,1cyclictest26272-21grep22:35:161
791299118,2cyclictest4897-21sed23:05:181
791299118,2cyclictest3545-21ssh22:41:011
791299118,2cyclictest29767-21sh23:22:251
791299118,2cyclictest28370-21rm21:29:101
791299118,2cyclictest26281-21ssh22:57:231
791299118,2cyclictest23912-21ssh23:18:511
791299118,2cyclictest21390-21grep19:35:191
791299118,2cyclictest18622-21ssh22:06:541
791299118,2cyclictest15856-21ssh22:50:171
791299118,2cyclictest15736-21tr21:20:211
791299118,2cyclictest14148-21wc19:20:201
791299118,2cyclictest11336-21chrt20:24:251
791299117,3cyclictest26939-21sed20:55:221
791299111,9cyclictest626-21diskmemload21:45:141
791299110,10cyclictest5186-21sort22:20:191
791099118,2cyclictest8205-21ssh22:22:130
791099118,2cyclictest4252-21basename23:50:170
791099118,2cyclictest32683-21ssh21:54:570
791099118,2cyclictest30636-21ssh22:38:030
791099118,2cyclictest25909-21appstreamcli00:30:010
791099118,2cyclictest25362-21ssh21:49:460
791099118,2cyclictest24951-21ssh22:56:070
791099118,2cyclictest24943-21ssh21:25:540
791099118,2cyclictest24405-21grep19:40:340
791099118,2cyclictest21350-21ssh22:09:280
791099118,2cyclictest21130-21ssh23:16:160
791099118,2cyclictest21108-21cron20:45:000
791099118,2cyclictest20046-21apache_accesses00:25:130
791099118,2cyclictest18622-21ssh22:30:030
791099118,2cyclictest17326-21ssh00:22:180
791099118,2cyclictest14814-21head20:30:180
791099118,2cyclictest13009-21tr20:25:250
791099118,2cyclictest1216-21cut23:25:170
791099118,2cyclictest11436-21expr23:55:160
7910991110,1cyclictest0-21swapper/023:35:380
791099110,10cyclictest8424-21expr23:30:180
791899108,1cyclictest31628-21taskset19:56:383
791899108,1cyclictest25354-21basename00:05:163
791899107,2cyclictest7995-21grep20:15:243
791599108,1cyclictest6888-21apt20:15:162
791599108,1cyclictest22511-21ssh21:25:152
791599107,2cyclictest18922-21ssh21:45:152
791599106,2cyclictest1333-21wc20:00:282
791299108,1cyclictest40362sleep120:07:301
791299108,1cyclictest29471-21ssh21:30:011
791299108,1cyclictest20785-21chrt20:40:511
791299108,1cyclictest18460-21taskset20:39:311
791299108,1cyclictest18173-21ls19:30:151
791299107,2cyclictest8748-21grep19:10:191
791299107,2cyclictest8748-21grep19:10:181
791299107,2cyclictest8684-21cut23:30:191
791299107,2cyclictest6894-21aten2_r7power_c20:15:171
791299107,2cyclictest512-21grep20:00:191
791299107,2cyclictest31126-21taskset23:00:271
791299107,2cyclictest25300-21sort20:50:291
791299107,2cyclictest24583-21taskset19:42:321
791299107,2cyclictest18951-21wget22:30:161
791299100,4cyclictest13578-21sshd20:27:401
791099108,1cyclictest32338-21egrep20:00:170
791099108,1cyclictest25190-21ssh00:05:080
791099108,1cyclictest16881-21expr20:35:160
791099108,1cyclictest11191-21cut19:15:180
791099107,2cyclictest26450-21tr20:55:190
791099107,2cyclictest25693-21grep19:45:220
791099107,1cyclictest15700-21cat19:25:160
791099102,4cyclictest224-21systemd-journal19:54:590
791099100,9cyclictest5155-21tune2fs20:10:200
79189996,2cyclictest22536-21tail20:45:263
79159997,1cyclictest326962sleep221:05:322
79159997,1cyclictest10819-21wc19:15:162
79129997,1cyclictest5110-21fschecks_count20:10:201
79129997,1cyclictest26003-21cut19:45:241
79129997,1cyclictest21541-21cut20:45:201
79129997,1cyclictest139952sleep120:30:051
79129996,2cyclictest10818-21ls19:15:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional