You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 15:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Dec 04, 2025 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
18528210287,10sleep10-21swapper/107:05:511
1671529986,9sleep30-21swapper/307:05:083
1867228168,9sleep00-21swapper/007:07:430
1875628070,6sleep20-21swapper/207:08:502
141742310,1sleep2311rcuc/210:06:242
18945992625,1cyclictest0-21swapper/310:40:423
1893699242,17cyclictest0-21swapper/011:25:450
18936992221,1cyclictest0-21swapper/011:39:250
1893699210,1cyclictest0-21swapper/012:20:140
1894199190,19cyclictest0-21swapper/211:09:532
1893999198,2cyclictest141rcu_preempt10:35:481
1894199182,1cyclictest12206-21diskmemload09:38:482
1894199180,13cyclictest0-21swapper/207:20:012
1893999186,11cyclictest7744-21snmpd09:25:131
1893699173,12cyclictest9261-21diskstats10:55:180
1893699170,2cyclictest0-21swapper/009:19:470
1893699170,17cyclictest0-21swapper/011:41:590
1893699170,17cyclictest0-21swapper/010:42:270
1894599164,9cyclictest9058-21cut09:35:173
1894599160,13cyclictest0-21swapper/309:45:143
1894199168,6cyclictest12744-21wc11:49:592
1894199162,3cyclictest141rcu_preempt11:26:122
18941991615,1cyclictest0-21swapper/209:30:042
18941991614,1cyclictest2778-21expr10:50:152
1893999168,4cyclictest224-21systemd-journal10:00:571
1893699161,8cyclictest5814-21munin-run07:49:590
235912150,0sleep10-21swapper/111:57:471
18945991514,1cyclictest0-21swapper/312:11:433
18945991514,1cyclictest0-21swapper/310:50:133
1894199152,3cyclictest224-21systemd-journal07:15:002
1894199150,15cyclictest0-21swapper/212:06:592
1894199150,14cyclictest28456-21sensors09:50:242
1893699150,2cyclictest0-21swapper/012:01:000
18945991410,3cyclictest9999-21sed12:35:283
1894599140,13cyclictest0-21swapper/312:06:233
1894199149,4cyclictest32668-21fgrep07:35:222
1894199149,3cyclictest26082-21apache212:32:092
18941991410,3cyclictest19259-21cron11:55:002
1894199140,1cyclictest0-21swapper/212:17:252
1894199140,1cyclictest0-21swapper/210:28:032
18939991412,1cyclictest10359-21uniq10:30:161
18939991411,2cyclictest7989-21grep07:50:271
18936991410,3cyclictest25285-21tr07:20:210
1893699140,14cyclictest0-21swapper/011:56:020
18945991313,0cyclictest12206-21diskmemload11:20:353
18945991311,1cyclictest11531-21tr08:00:163
18945991310,2cyclictest6169-21cron11:45:003
18945991310,2cyclictest20397-21head07:10:213
18945991310,2cyclictest20216-21grep11:30:193
1894199139,3cyclictest27006-21grep12:00:172
1894199139,3cyclictest26366-21tail08:30:172
1894199135,8cyclictest0-21swapper/210:33:112
18941991312,1cyclictest0-21swapper/210:55:242
18941991310,2cyclictest5227-21ls10:00:002
18941991310,2cyclictest30302-21tail10:45:272
18941991310,2cyclictest15836-21wc08:09:592
1894199130,3cyclictest0-21swapper/211:20:162
1893999132,3cyclictest0-21swapper/112:04:541
18939991310,2cyclictest526-21sed07:35:251
1893999130,12cyclictest0-21swapper/111:03:451
1893699139,3cyclictest1739-21cat11:15:230
1893699136,6cyclictest14343-21munin-run12:40:000
1893699131,6cyclictest8423-21idleruntime-cro07:55:000
18936991311,1cyclictest0-21swapper/012:29:090
18936991311,1cyclictest0-21swapper/012:29:090
18936991310,2cyclictest907-21rm09:27:060
18936991310,2cyclictest27121-21grep11:10:140
18936991310,2cyclictest21817-21tr07:15:170
18936991310,2cyclictest16175-21tr10:35:180
18936991310,2cyclictest12005-21grep08:00:180
1893699130,4cyclictest0-21swapper/011:20:160
1893699130,2cyclictest0-21swapper/011:45:550
1893699130,1cyclictest0-21swapper/012:14:230
1894599129,2cyclictest3702-21tr09:30:173
1894599129,2cyclictest15321-21expr11:00:173
1894599129,2cyclictest14205-21df08:05:163
1894599129,2cyclictest1394-21cut07:40:163
1894599129,2cyclictest13598-21grep11:50:153
1894599129,2cyclictest13280-21basename12:15:153
1894599128,3cyclictest29229-21grep08:35:213
18945991210,1cyclictest7251-21cat10:00:243
18945991210,1cyclictest28731-21grep10:45:153
18945991210,1cyclictest21577-21sed08:20:173
18945991210,1cyclictest19164-21grep08:15:183
1894199129,2cyclictest9499-21head12:35:222
1894199129,2cyclictest5822-21grep07:49:592
1894199129,2cyclictest31362-21sort10:20:292
1894199129,2cyclictest28839-21grep12:25:292
1894199129,2cyclictest28839-21grep12:25:282
1894199129,2cyclictest23143-21cut10:40:182
1894199129,2cyclictest22075-21wc08:20:222
1894199129,2cyclictest21226-21grep11:30:282
1894199129,2cyclictest20056-21tail08:15:272
1894199128,3cyclictest21894-21head12:20:282
1894199128,3cyclictest12894-21tr09:10:152
1894199128,3cyclictest10383-21tr09:05:212
1894199124,4cyclictest224-21systemd-journal09:24:562
1894199120,12cyclictest0-21swapper/211:40:152
1894199120,11cyclictest0-21swapper/211:39:422
1893999129,2cyclictest8350-21sed12:10:271
1893999129,2cyclictest4433-21grep07:45:151
1893999129,2cyclictest4055-21ssh09:30:181
1893999129,2cyclictest32059-21wc11:39:591
1893999129,2cyclictest3001-21tr08:50:171
1893999129,2cyclictest29629-21sed08:35:251
1893999129,2cyclictest25791-21tr08:30:151
1893999129,2cyclictest2551-21ntpq12:30:231
1893999129,2cyclictest24895-21tail07:20:181
1893999129,2cyclictest17846-21tr10:10:161
1893999129,2cyclictest16652-21cut08:10:161
1893999129,2cyclictest15359-21grep08:05:271
1893999129,2cyclictest12206-21diskmemload10:53:551
1893999128,3cyclictest28701-21sh07:30:011
1893999128,2cyclictest13898-21tail11:25:181
1893999127,4cyclictest26082-21apache211:32:211
1893999121,3cyclictest0-21swapper/110:40:121
18939991210,1cyclictest1325-21apt-get07:40:161
1893999120,11cyclictest29204-21cut07:30:151
1893699129,2cyclictest8939-21grep09:00:210
1893699129,2cyclictest7367-21grep10:00:240
1893699129,2cyclictest5970-21idleruntime-cro12:10:000
1893699129,2cyclictest30411-21head10:20:200
1893699129,2cyclictest29007-21grep07:30:140
1893699129,2cyclictest24382-21ls08:25:200
1893699129,2cyclictest21853-21grep11:05:170
1893699129,2cyclictest20949-21grep09:45:170
1893699129,2cyclictest19658-21grep11:30:170
1893699129,2cyclictest16761-21grep09:40:260
1893699128,3cyclictest30339-21head10:45:270
1893699120,9cyclictest0-21swapper/009:55:150
1893699120,12cyclictest0-21swapper/010:13:220
1893699120,11cyclictest27752-21head09:50:180
1894599119,1cyclictest8476-21sed09:00:173
1894599119,1cyclictest8391-21cron07:55:003
1894599119,1cyclictest6047-21cat07:50:003
1894599119,1cyclictest20007-21grep11:55:143
1894599119,1cyclictest15758-21head09:40:183
1894599118,2cyclictest8418-21grep11:45:273
1894599118,2cyclictest341-21ssh11:15:013
1894599118,2cyclictest30839-21uname10:20:233
1894599118,2cyclictest30677-21tr08:40:183
1894599118,2cyclictest2785-21ssh09:29:453
1894599118,2cyclictest27718-21grep12:25:193
1894599118,2cyclictest27718-21grep12:25:183
1894599118,2cyclictest26849-21grep08:30:213
1894599118,2cyclictest25334-21ssh12:23:513
1894599118,2cyclictest2043-21ssh09:55:403
1894599118,2cyclictest18866-21ssh09:15:153
1894599118,2cyclictest17389-21ssh10:10:003
1894599118,2cyclictest15333-21sh10:07:533
1894599118,2cyclictest1242-21basename12:30:133
1894599117,2cyclictest29684-21ssh11:11:293
1894599113,6cyclictest8883-21apt-get07:55:123
1894599110,11cyclictest0-21swapper/310:55:173
1894599110,10cyclictest31648-21cat07:35:153
1894599110,10cyclictest0-21swapper/311:26:033
1894199118,2cyclictest9685-21taskset09:03:212
1894199118,2cyclictest7037-21grep12:10:162
1894199118,2cyclictest7017-21taskset08:57:372
1894199118,2cyclictest31137-21cat07:35:002
1894199118,2cyclictest3079-21chrt07:40:342
1894199118,2cyclictest2990-21cron09:30:002
1894199118,2cyclictest27214-21sh11:10:162
1894199118,2cyclictest26505-21sleep09:49:592
1894199118,2cyclictest19543-21head09:15:182
1894199118,2cyclictest16414-21cut09:40:232
1894199113,3cyclictest964-21fschecks_count08:45:162
1894199111,7cyclictest0-21swapper/210:00:102
1894199110,10cyclictest21228-21ssh10:13:012
1893999118,2cyclictest8738-21sh09:35:061
1893999118,2cyclictest7993-21tr09:00:141
1893999118,2cyclictest7635-21tail11:20:201
1893999118,2cyclictest7272-21cut11:45:181
1893999118,2cyclictest5874-21cut08:55:251
1893999118,2cyclictest5508-21ssh11:44:171
1893999118,2cyclictest3781-21grep10:25:141
1893999118,2cyclictest3707-21sh09:58:151
1893999118,2cyclictest32363-21sh12:05:031
1893999118,2cyclictest30744-21ssh10:20:231
1893999118,2cyclictest30445-21cut08:40:161
1893999118,2cyclictest29720-21ssh10:45:221
1893999118,2cyclictest27764-21cut09:50:181
1893999118,2cyclictest27657-21cut11:10:181
1893999118,2cyclictest24884-21grep08:25:261
1893999118,2cyclictest23812-21expr10:15:151
1893999118,2cyclictest22874-21wc07:15:241
1893999118,2cyclictest20836-21chrt09:15:371
1893999118,2cyclictest19791-21cut07:10:161
1893999118,2cyclictest19042-21cut08:15:181
1893999118,2cyclictest16674-21chrt09:13:121
1893999118,2cyclictest16101-21sshd11:51:261
1893999118,2cyclictest15419-21sh09:40:161
1893999118,2cyclictest14522-21sed12:15:211
1893999118,2cyclictest13051-21ssh10:05:251
1893999118,2cyclictest11709-21sh12:37:021
1893999116,2cyclictest224-21systemd-journal08:24:591
1893999110,10cyclictest22601-21ssh11:05:251
1893699118,2cyclictest8745-21ssh09:35:060
1893699118,2cyclictest4550-21grep10:25:190
1893699118,2cyclictest4274-21wc08:50:290
1893699118,2cyclictest29994-21chrt08:38:370
1893699118,2cyclictest26946-21ssh09:20:450
1893699118,2cyclictest26158-21taskset10:16:220
1893699118,2cyclictest18629-21ssh11:54:130
1893699118,2cyclictest15758-21cron08:10:000
1893699118,2cyclictest14402-21ntpq12:15:200
1893699118,2cyclictest13543-21taskset10:05:360
1893699118,2cyclictest12597-21ssh09:10:110
1893699118,2cyclictest1086-21sh12:30:030
1893699117,3cyclictest19929-21sed07:10:170
1893699110,10cyclictest7203-21sh10:53:590
1893699110,10cyclictest5140-21cat09:30:280
1894599108,2cyclictest4875-21cut08:55:013
1894599108,2cyclictest27538-21cat11:35:273
1894599108,1cyclictest9818-21ssh10:30:153
1894599108,1cyclictest28482-21python09:50:263
1894599108,1cyclictest28040-21ssh09:22:053
1894599108,1cyclictest26405-21ssh11:09:433
1894599108,1cyclictest24884-21head07:20:183
1894599108,1cyclictest23278-21ssh10:15:093
1894599108,1cyclictest23147-21chrt08:25:033
1894599108,1cyclictest15747-21rm10:35:103
1894599108,1cyclictest12515-21ssh09:10:053
1894599107,2cyclictest29119-21ssh12:01:293
1894599107,2cyclictest16240-21cat08:10:163
1894599101,8cyclictest0-21swapper/310:25:143
1894199108,1cyclictest4657-21chrt08:52:512
1894199108,1cyclictest30837-21cat08:40:182
1894199108,1cyclictest27067-21expr07:25:172
1894199108,1cyclictest25881-21ssh10:16:072
1894199108,1cyclictest20038-21ssh10:38:032
1894199108,1cyclictest11232-21taskset08:00:032
1894199108,1cyclictest10609-21who07:55:252
1894199108,1cyclictest0-21swapper/211:55:202
1894199107,2cyclictest29067-21cat08:35:192
1894199107,2cyclictest17909-21chrt08:12:072
1894199107,2cyclictest17473-21ssh11:01:112
1894199107,2cyclictest1613-21uname11:15:222
1894199106,3cyclictest7417-21tr07:50:212
1894199100,9cyclictest261882sleep207:23:182
1894199100,9cyclictest24125-21tr08:25:182
1893999108,1cyclictest26360-21smart_sda09:20:261
1893999108,1cyclictest24022sleep108:49:241
1893999107,2cyclictest8446-21sh10:55:031
1893999107,2cyclictest582-21ssh11:15:151
1893999107,2cyclictest21852-21cut09:45:221
1893999107,2cyclictest20317-21grep12:20:151
1893999107,2cyclictest12539-21wc08:00:231
1893999106,3cyclictest12319-21wc09:10:011
1893699108,1cyclictest9439-21ssh10:30:010
1893699108,1cyclictest9015-21wget07:55:140
1893699108,1cyclictest7262sleep007:36:150
1893699108,1cyclictest31195-21which08:40:200
1893699108,1cyclictest26757-21ntpq08:30:210
1893699108,1cyclictest2461-21ntpq07:40:230
1893699108,1cyclictest15472-21expr11:00:170
1893699108,1cyclictest11049-21cut09:05:250
1893699107,2cyclictest5829-21grep08:55:250
1893699107,2cyclictest27986-21sensors07:25:230
1893699107,2cyclictest21364-21grep08:20:150
1893699107,2cyclictest18878-21cat08:15:160
1893699107,2cyclictest16634-21sort08:10:160
189459998,1cyclictest692-21expr08:45:143
189459998,1cyclictest5918-21hostname08:55:243
189459998,1cyclictest30990-21chrt07:33:573
189459997,1cyclictest26552-21seq07:25:023
189459997,1cyclictest21699-21grep07:15:163
189459997,1cyclictest11721-21grep09:05:323
189399997,1cyclictest26881-21ssh12:25:101
189399997,1cyclictest26881-21ssh12:25:091
189399997,1cyclictest10407-21sed07:55:241
189369997,1cyclictest32732-21chrt08:45:020
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional