You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-15 - 06:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Wed Apr 15, 2026 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
718499785,49cyclictest16923-21ssh23:10:130
7198995913,8cyclictest131rcu_preempt20:00:121
7218995746,7cyclictest35966irq/58-eth022:15:153
7210995334,13cyclictest7557-21ssh22:45:242
7198995312,18cyclictest21-21ksoftirqd/100:35:101
718499503,8cyclictest12-21ksoftirqd/000:20:120
7184995030,5cyclictest12-21ksoftirqd/023:20:130
721099484,33cyclictest27364-21ssh22:40:152
7218994718,20cyclictest35966irq/58-eth023:15:163
721099440,30cyclictest26159-21ssh23:15:162
719899445,36cyclictest6453-21ssh23:40:141
721899434,26cyclictest35966irq/58-eth021:45:143
719899423,32cyclictest10342-21ssh00:00:141
7198994211,18cyclictest21-21ksoftirqd/100:10:151
7198994211,18cyclictest21-21ksoftirqd/100:10:151
718499422,30cyclictest5083-21ssh00:15:130
719899411,31cyclictest3984-21ssh22:45:141
718499401,5cyclictest131rcu_preempt21:30:150
721899393,23cyclictest35966irq/58-eth022:50:123
721899381,23cyclictest35966irq/58-eth023:40:123
721899371,21cyclictest35966irq/58-eth000:10:133
721899371,21cyclictest35966irq/58-eth000:10:123
7198993720,11cyclictest21-21ksoftirqd/100:05:161
718499373,30cyclictest9239-21ssh22:30:140
718499373,29cyclictest11355-21ssh21:20:120
721899365,17cyclictest35966irq/58-eth023:35:133
7210993617,13cyclictest27346irq/40-dwc2_hso22:10:112
7210993613,19cyclictest131rcu_preempt00:35:112
719899365,24cyclictest14329-21ssh22:15:151
7218993521,9cyclictest35966irq/58-eth022:00:133
7218993515,11cyclictest321rcuc/320:50:133
719899353,21cyclictest21-21ksoftirqd/119:25:141
7198993524,7cyclictest25178-21diskmemload22:25:131
7198993524,7cyclictest25178-21diskmemload22:25:121
718499351,23cyclictest13062-21sh22:50:140
721099342,28cyclictest32524-21ssh22:25:132
721099342,28cyclictest32524-21ssh22:25:132
719899344,20cyclictest11407-21kworker/u8:122:05:121
7184993421,5cyclictest12-21ksoftirqd/020:45:140
721099332,18cyclictest27346irq/40-dwc2_hso23:20:152
719899331,23cyclictest2249-21ssh21:15:151
718499316,19cyclictest12-21ksoftirqd/022:15:140
718499312,22cyclictest5164-21ssh22:10:110
721099302,21cyclictest19162-21ntp_192.168.11423:45:262
721099302,11cyclictest24504-21ntp_states23:30:262
721099293,20cyclictest1263-21ssh23:55:152
721099293,11cyclictest0-21swapper/221:27:242
721099292,16cyclictest27346irq/40-dwc2_hso20:40:152
719899292,20cyclictest3864-21rm23:37:241
719899292,20cyclictest26850-21ntpq00:25:251
719899291,21cyclictest6212-21rm21:35:141
7198992910,11cyclictest7174-21cyclictest00:30:141
718499291,25cyclictest20459-21ssh21:25:130
7184992910,13cyclictest12-21ksoftirqd/019:10:130
721099284,14cyclictest27346irq/40-dwc2_hso22:20:132
721099284,13cyclictest27346irq/40-dwc2_hso19:30:132
721099283,19cyclictest27346irq/40-dwc2_hso19:40:142
721099283,14cyclictest25598-21ssh21:10:152
721099282,10cyclictest0-21swapper/223:37:252
7210992820,4cyclictest27-21ksoftirqd/223:40:132
721099281,21cyclictest28454-21ssh00:10:152
721099281,21cyclictest28454-21ssh00:10:142
721099281,16cyclictest11775-21ssh23:25:122
719899283,20cyclictest10219-21ssh21:55:141
719899282,21cyclictest19374-21ssh22:00:131
7198992812,11cyclictest21-21ksoftirqd/123:32:251
721899273,18cyclictest0-21swapper/300:04:263
721099274,8cyclictest0-21swapper/222:35:132
721099274,7cyclictest24599-21ssh23:50:112
721099274,18cyclictest0-21swapper/200:20:142
721099272,17cyclictest27346irq/40-dwc2_hso20:35:142
721099272,10cyclictest0-21swapper/220:00:132
721099271,21cyclictest25027-21latency19:25:222
7210992711,10cyclictest0-21swapper/221:55:132
719899272,19cyclictest29038-21ssh22:21:251
721899261,17cyclictest0-21swapper/322:45:143
721099263,19cyclictest27-21ksoftirqd/223:10:142
721099263,16cyclictest27346irq/40-dwc2_hso00:25:132
721099261,20cyclictest1152-21ssh21:50:142
721099261,12cyclictest6308-21ssh21:35:142
719899267,11cyclictest7174-21cyclictest20:40:131
719899263,17cyclictest22137-21ssh22:55:141
719899262,18cyclictest24118-21ssh21:25:261
7198992614,3cyclictest21-21ksoftirqd/122:10:071
7218992512,6cyclictest0-21swapper/321:10:143
721099251,13cyclictest27346irq/40-dwc2_hso19:20:132
719899251,17cyclictest27363-21sh22:40:141
718499255,16cyclictest29843-21ssh23:35:120
718499254,14cyclictest20791-21ssh23:30:110
718499254,13cyclictest0-21swapper/020:05:120
721899243,13cyclictest0-21swapper/322:21:243
721899243,13cyclictest0-21swapper/321:35:133
7218992416,5cyclictest35966irq/58-eth022:05:143
721099242,18cyclictest17314-21ssh21:21:512
721099241,8cyclictest16004-21ssh23:09:262
719899247,8cyclictest21-21ksoftirqd/123:48:561
719899242,18cyclictest24497-21ssh21:45:141
719899242,17cyclictest25178-21diskmemload23:29:251
719899241,19cyclictest7612-21ls23:05:021
718499241,17cyclictest11802-21ssh23:25:140
721899233,14cyclictest0-21swapper/321:25:263
721899233,13cyclictest0-21swapper/321:20:253
721899233,11cyclictest0-21swapper/321:55:143
721899232,10cyclictest0-21swapper/323:58:243
721099233,15cyclictest0-21swapper/222:16:552
7210992312,9cyclictest30534-21ssh21:46:552
721099231,18cyclictest25178-21diskmemload23:03:252
7198992315,5cyclictest9925-21sh00:00:001
7198992311,8cyclictest21-21ksoftirqd/120:10:121
719899231,17cyclictest13327-21find19:50:111
719899231,17cyclictest13327-21find19:50:101
718499238,10cyclictest18297-21sh22:35:120
718499233,15cyclictest844-21munin-node22:25:140
718499233,15cyclictest844-21munin-node22:25:130
718499233,14cyclictest0-21swapper/020:15:130
7218992214,6cyclictest35966irq/58-eth022:10:213
7218992214,5cyclictest35966irq/58-eth021:40:123
721099223,15cyclictest27346irq/40-dwc2_hso19:35:132
721099222,16cyclictest0-21swapper/222:55:132
721099222,15cyclictest0-21swapper/222:53:262
721099222,15cyclictest0-21swapper/200:18:262
721099222,14cyclictest27346irq/40-dwc2_hso19:15:132
7210992211,9cyclictest26655-21ssh22:03:212
721099221,13cyclictest8606-21kthreadcore19:10:192
7210992210,9cyclictest0-21swapper/220:50:162
719899227,4cyclictest21-21ksoftirqd/121:24:491
719899222,5cyclictest0-21swapper/122:33:251
719899222,16cyclictest12108-21apt-get19:15:141
719899220,19cyclictest2759-21find23:20:131
718499228,4cyclictest131rcu_preempt23:55:010
7184992215,4cyclictest12-21ksoftirqd/021:45:140
718499221,17cyclictest32424-21ls22:05:290
718499221,16cyclictest6208-21sh21:35:140
721899213,16cyclictest33-21ksoftirqd/323:27:463
721899213,12cyclictest0-21swapper/300:30:143
721899212,13cyclictest0-21swapper/322:40:153
7218992114,3cyclictest35966irq/58-eth000:20:143
721899211,13cyclictest35966irq/58-eth019:50:123
721899211,13cyclictest35966irq/58-eth019:50:113
721099214,10cyclictest27346irq/40-dwc2_hso21:40:132
721099213,4cyclictest0-21swapper/220:55:122
721099213,10cyclictest29824-21hwlatdetect21:30:192
721099211,16cyclictest19157-21latency_hist00:05:012
721099211,16cyclictest13112-21idleruntime-cro19:50:002
721099211,16cyclictest11718-21cut22:30:232
721099211,10cyclictest30625-21kthreadcore22:05:202
7210992110,9cyclictest4752-21sshd20:14:252
721099210,12cyclictest3222966ssh00:30:092
7198992112,8cyclictest21-21ksoftirqd/122:50:121
719899211,18cyclictest18303-21ls20:25:291
718499217,5cyclictest131rcu_preempt19:30:120
718499214,9cyclictest0-21swapper/000:00:140
718499213,3cyclictest0-21swapper/021:15:160
721899207,10cyclictest0-21swapper/321:50:193
7218992014,4cyclictest0-21swapper/300:05:263
7218992011,7cyclictest0-21swapper/300:37:163
721899201,15cyclictest35966irq/58-eth023:05:143
721899201,12cyclictest0-21swapper/323:00:123
721099203,8cyclictest19821-21ssh00:05:202
721099203,14cyclictest23523-21kthreadcore21:05:232
721099201,9cyclictest4875-21kthreadcore21:15:232
721099201,11cyclictest15827-21apt-get21:00:142
721099201,11cyclictest10337-21sed20:20:172
721099201,10cyclictest19574-21fschecks_time20:30:152
719899208,3cyclictest131rcu_preempt20:50:021
719899207,4cyclictest21-21ksoftirqd/123:11:001
719899201,3cyclictest131rcu_preempt21:32:491
7198992012,6cyclictest0-21swapper/121:41:591
719899201,17cyclictest1125-21apt-get21:50:121
719899201,15cyclictest24870-21ls23:50:151
719899201,11cyclictest0-21swapper/120:21:321
718499209,8cyclictest1264-21diskstats21:50:160
718499207,5cyclictest131rcu_preempt22:55:130
718499206,4cyclictest131rcu_preempt00:30:180
718499203,12cyclictest0-21swapper/021:05:150
718499202,13cyclictest19365-21ssh22:00:130
718499201,7cyclictest28715-21hwlatdetect00:10:180
718499201,7cyclictest28715-21hwlatdetect00:10:170
721899198,9cyclictest0-21swapper/321:10:023
721899196,6cyclictest35966irq/58-eth000:15:113
721899193,11cyclictest0-21swapper/300:28:253
7218991911,6cyclictest35966irq/58-eth023:10:133
7218991911,6cyclictest0-21swapper/322:31:073
721899191,13cyclictest0-21swapper/323:45:183
721899191,12cyclictest0-21swapper/319:30:273
721099193,8cyclictest1746-21irqcore20:45:192
721099193,13cyclictest7230-21latency_hist19:10:002
721099192,9cyclictest29552-21kthreadcore20:05:202
721099192,13cyclictest15517-21kthreadcore20:25:202
721099191,9cyclictest8327-21missed_timers20:15:242
721099191,9cyclictest18299-21tune2fs19:55:162
721099191,15cyclictest13561-21wc19:50:162
721099191,15cyclictest13561-21wc19:50:152
719899198,8cyclictest5071-21apt-get00:15:121
719899198,8cyclictest21873-21ntp_192.168.11422:35:241
719899196,4cyclictest21-21ksoftirqd/121:10:161
719899195,8cyclictest0-21swapper/120:35:151
719899192,15cyclictest28169-21kthreadcore19:30:201
7198991911,5cyclictest0-21swapper/120:30:221
719899191,13cyclictest26150-21ssh23:15:141
719899191,10cyclictest7834-21/usr/sbin/munin23:05:281
7198991910,5cyclictest0-21swapper/100:25:021
718499197,9cyclictest18997-21cat22:00:020
718499197,3cyclictest131rcu_preempt23:00:140
718499196,10cyclictest27513-21fschecks_count22:40:170
718499194,8cyclictest0-21swapper/023:05:120
718499192,11cyclictest19053-21ssh23:45:240
718499191,14cyclictest31369-21apt-get19:35:140
718499191,14cyclictest23451-21sh22:20:140
721899189,7cyclictest0-21swapper/323:30:583
721899189,7cyclictest0-21swapper/322:58:163
721899189,7cyclictest0-21swapper/321:18:523
721899189,6cyclictest35966irq/58-eth023:20:123
721899183,11cyclictest0-21swapper/322:25:133
721899183,11cyclictest0-21swapper/322:25:123
721899183,10cyclictest35966irq/58-eth023:50:233
721899182,10cyclictest0-21swapper/320:40:153
7218991811,5cyclictest35966irq/58-eth022:35:123
719899188,7cyclictest8710-21grep19:45:171
719899185,3cyclictest21-21ksoftirqd/120:05:271
7198991811,4cyclictest17260-21grep19:20:171
718499189,6cyclictest0-21swapper/023:45:010
718499186,9cyclictest23235-21ssh00:25:140
718499186,9cyclictest19183-21cron20:30:000
7184991812,4cyclictest14341-21missed_timers20:55:230
718499181,13cyclictest19432-21ssh00:05:140
718499181,13cyclictest15333-21ssh21:40:120
718499181,11cyclictest26330-21latency_hist19:30:020
721899177,8cyclictest33-21ksoftirqd/321:30:143
719899174,4cyclictest21-21ksoftirqd/120:50:271
719899172,10cyclictest0-21swapper/119:40:151
7198991711,4cyclictest0-21swapper/119:10:261
719899171,13cyclictest21753-21switchtime19:55:261
719899171,11cyclictest7454-21cut20:15:211
718499179,6cyclictest4741-21sshd20:11:360
718499179,5cyclictest4374-21irqrtprio19:40:180
718499178,7cyclictest4093-21df22:45:150
718499178,7cyclictest10256-21cut00:35:190
718499172,4cyclictest9924-21munin-run00:00:010
718499172,10cyclictest0-21swapper/020:50:150
718499171,6cyclictest26263-21df_inode23:15:170
718499171,10cyclictest15791-21timerandwakeup19:15:280
721899163,8cyclictest0-21swapper/320:35:153
721899162,9cyclictest0-21swapper/319:55:133
721899162,9cyclictest0-21swapper/319:45:153
721899161,12cyclictest0-21swapper/320:15:233
721899161,12cyclictest0-21swapper/319:20:163
719899167,5cyclictest25175-21cron21:10:021
719899162,10cyclictest0-21swapper/119:35:151
719899161,11cyclictest7275-21latency_hist19:10:021
719899161,11cyclictest18974-21kthreadcore21:00:221
719899161,11cyclictest11002-21apt-get20:55:141
718499169,4cyclictest25599-21cpu21:10:150
718499168,3cyclictest12304-21timerwakeupswit19:45:290
718499163,5cyclictest22928-21apt-get20:00:120
718499163,11cyclictest17073-21fschecks_count19:20:160
718499163,10cyclictest14012-21kthreadcore19:50:180
718499163,10cyclictest14012-21kthreadcore19:50:180
718499162,9cyclictest0-21swapper/021:00:150
718499162,10cyclictest0-21swapper/020:20:140
718499161,12cyclictest21414-21kthreadcore20:30:190
718499161,11cyclictest32668-21switchtime20:40:290
718499161,11cyclictest24433-21grep20:35:170
718499160,9cyclictest18205-21cpuspeed19:55:130
721899152,9cyclictest0-21swapper/319:35:153
721899147,3cyclictest0-21swapper/320:25:123
721899143,8cyclictest35966irq/58-eth020:45:143
721899142,9cyclictest0-21swapper/320:10:143
721899142,9cyclictest0-21swapper/320:10:013
721899142,7cyclictest0-21swapper/320:00:123
721899142,10cyclictest0-21swapper/321:00:143
721899141,8cyclictest0-21swapper/321:00:023
721899141,10cyclictest0-21swapper/320:20:163
718499141,10cyclictest7281-21latency_hist19:10:010
721899138,3cyclictest0-21swapper/320:30:253
721899138,3cyclictest0-21swapper/319:15:143
721899132,9cyclictest0-21swapper/319:25:253
721899131,8cyclictest0-21swapper/319:40:143
721899131,7cyclictest0-21swapper/319:10:273
721899111,7cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional